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Executive Summary 

Over the last 20 years, state education agencies (SEAs) have been given considerably more 
responsibilities for directing and guiding the improvement of low-performing schools. At the 
same time, federal policies strongly pressed SEAs to use research to design these supports. Very 
few studies have explored the SEA as an organization, or its role in accessing and using research. 
Likewise, few, if any, have studied the role of social networks in the organization and flow of 
information in SEAs. This exploratory study was designed to fill those gaps by examining where 
and how a purposive sample of three SEAs searched for, incorporated, and used research and 
other types of knowledge to design, implement, and refine state school improvement policies, 
programs and practices. 

The staff in these three SEAs actively sought and were receptive to research. Contrary to a 
uni-dimensional model of knowledge utilization, where research users are viewed as passive 
recipients of published research, research use was a multi-dimensional process in these sites. 
Multiple SEA staff reached out to numerous internal and external sources for research as well 
as other types of knowledge on school improvement. Incorporating research into policy and 
practice was often a social process, where SEA staff worked with each other, practitioners, and 
external partners to make sense of the research and adapt it to their local context. Key brokers of 
research inside and outside the SEAs facilitated the research search and incorporation process. 
Contrary to the usual image of SEAs as siloed organizations, we found considerable cross-office 
and department communication in the work and research networks. 

Decision-makers were more likely to seek and use "research designed for use," that is, tools 
or strategies that provided detailed guidance for action and were derived from research, than 
individual published academic studies. SEA staff also understood that research, particularly from 
recognizable and trusted sources, lent credibility to their efforts and motivated practitioners. 
Although SEAs turned to external organizations less often, in two states these sources were 
critical in providing access to research and in synthesizing and translating it into useful guides 
for action. 

SEAs can strengthen research-based knowledge use by: 1) reaching out more often to external 
organizations, such as technical assistance centers, state and national professional membership 
organizations, and universities, which have ready access to research and research designed for 
use; 2) identifying, connecting and nurturing knowledge brokers in their agencies and outside 
organizations who work on common problems; and 3) fostering working groups composed of 
these brokers, key research sources, and practitioners to adapt generalized findings into more 
useable information. Policymakers should encourage and support SEA evaluations, particularly 
of the implementation of their own programs, to provide critical, systematic feedback to agency 
staff. Studies are also needed of how SEAs access and use research in other education policy 
areas, of the quality of research acquired by SEA staff and underlying research designed for use 
products, and on policy implementation and effects in understudied areas of education policy. 
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CHAPTER 1 

Introduction 


Over the last two decades, state and federal laws and grant programs, such as state accountability 
polices, the No Child Left Behind Act (NCLB), Race to the Top, and Title I School Improvement 
Grants, have given state education agencies (SEAs) considerably more responsibilities for 
directing and guiding the improvement of low-performing schools. They have also increased 
the incentives and supports for SEAs and school districts to adopt research-based school 
improvement policies and practices. Title I, or other sections of the Elementary and Secondary 
Education Act(ESEA), and initiatives like the What Works Clearinghouse require, or in other ways 
press, SEAs to use research-based knowledge (RBK) in their statewide systems of support for 
low-performing schools, technical assistance for districts, professional developmentforteachers, 
and school improvement programs. 

Policymakers have urged SEAs to engage with other external intermediary organizations to 
extend their strained capacity to provide improvement supports for schools and districts, and 
to help them collect and use research or other evidence (Redding & Wahlberg 2008; Rennie 
Center, 2004). Many types of organizations are now involved in this enterprise, including for 
example, professional membership associations like the Council of Chief State School Officers, 
national or state-level research organizations like Mass Insight, and the large and complex 
federal system of technical assistance centers, among others. 

Although studies exist of districts' and schools' use of research (see for example, Clifford, 
Kochanek, Perez-Lopez, Behrstock-Sherratt, Meyers, Brown-Sims, & Fetters, 201 2; Coburn, Honig, 
& Stein, 2009; Coburn, Toure, & Yamashita, 2009; Daly & Finnigan, 2011; Farley-Ripple, 201 2), 
we know little about how SEAs search for, select, and use research and other kinds of evidence 
in their school improvement strategies. While one might assume similarities in research use 
behaviors, both the organizational structures of SEAs and the population of intermediaries 
with which they interact are quite different than schools and districts, and the most recent in- 
depth study of SEAs was conducted nearly 20 years ago (Lusi, 1 997). This exploratory study was 
designed to fill that gap by investigating three questions in a purposive sample of three SEAs: 


3 



State Education Agencies' Acquisition and Use of Research Knowledge in School Improvement Strategies 


1. How are SEAs organized for managing and using RBK in their improvement 
strategies for low-performing schools and school districts? 

2. How do SEAs search for, obtain, and incorporate RBK in policy and practices to 
support improvementfor low-performing schools and school districts? 

3. What role does RBK play in states' strategies to improve school districts and schools? 

Conceptual Framework 

The conceptual framework underlying our study is drawn from several lines of research and 
theory, including knowledge utilization studies, organization literature, the literature on social 
capital and social networks, school improvement research, and policy and evaluation research. 
These literatures led us to focus on five major areas of potential influence in the process of 
knowledge search, incorporation, and use: 1 ) the sources of knowledge; 2) the types and forms 
of knowledge exchanged within and between SEAs and intermediary organizations; 3) SEAs 
organizational structures and social networks; 4) knowledge search and incorporation processes; 
and, 5) political, contextual, and institutional factors (Figure 1 .1 ). We briefly summarize relevant 
findings from these literatures below; expanded reviews of the literature on organizational 
structures and networks, and search and incorporation, are included in Chapters 3 and 4, 
respectively. 

Sources of knowledge 

We focused on where SEA staff searched for knowledge, particularly the extent to which 
staff turned to internal sources within their own SEA for advice, or to external intermediary 
organizations. Studies in other settings have shown that educators typically seek out colleagues 
inside their organization for advice, particularly those who are organizationally or physically 
closest to them (see, for example, Supovitz & Weinbaum, 2008). SEAs have been portrayed 
as highly fragmented, "siloed" organizations with little communication across different units 
and programs (see Organizational Structures and Social Networks below). Thus, we considered 
the extent to which individuals sought information within their own offices, across offices and 
divisions of the SEA, or from regional education agencies, a level of organization above the 
districts and below SEAs. 

Research suggests that external organizations can be key conduits for research-based knowledge 
and for the effective translation of RBK into local practices. A variety of external organizations 
focused on the business of school improvement have grown considerably over the past several 
decades, providing information, training, materials, and programs.The federal government has 
targeted resources on helping SEAs develop a system of supports through its comprehensive 
assistance centers and other sponsored centers (Massed, Sallom, Perrault, Barnes, & Rowan, 
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2010; Turnbull, White, Sinclair, Riley, & Pistorino, 2011). Other intermediary organizations 
have become involved in this enterprise, including professional membership associations, 
national or state-level research and provider organizations, and institutions of higher education 
(Honig & Coburn, 2008; Rowan, 2002). We considered whether and to whom SEA staff turn for 
information within this array of potential outside sources. 

Types and forms of knowledge 

Many studies of knowledge use have not distinguished among different types of evidence. 
To address this issue, we identified three types of knowledge: research-based knowledge 
(RBK), other evidence-based knowledge (EBK), and practitioner knowledge. We define RBK as 
research findings that have been to varying degrees "collated, summarized, and synthesized," 
then presented in ways that provide "empirical or theoretical insights or make them otherwise 
informative" (Davies & Nutley, 2008). 1 We include in this category published original research, 
research syntheses, summaries or meta-analyses, and evaluation reports. These may be 
presented in journal articles, books, materials, or other media developed by intermediaries. 
We distinguish these more prevalent forms of RBK from forms that are designed for use in 
practice; that is, models, programs, protocols or other tools that embed research-based practices 
in guides to action. 

The school improvement literature indicates that certain characteristics of RBK designed for 
use can be important to whether and how well practitioners use it, especially the degree of 
specification (the detail with which school improvement policy, program or guidance elaborates 
action) and scaffolding (supports for guiding the use of knowledge in practice) provided to those 
meant to act upon the findings (Cohen & Ball, 2007; Cohen, Peurach, Glazer, Gates, & Goldin, 
2013). School improvement and leadership studies suggest that RBK that is designed for use 
with a relatively higherdegree of specificity and scaffolding is more likelyto be incorporated into 
educational practices, and with less variation. These models were shown to improve teaching, 
learning, and achievement in struggling elementary schools (see, for example, Borman, Hewes, 
Overman, & Brown, 2003; May, Supovitz, & Perda, 2004; Rowan, Correnti, Miller, & Camburn, 
2009). 

Older knowledge utilization models assumed that simply transmitting such knowledge to 
policymakers or practitioners would be sufficientto create change. But new modelsfor knowledge 
diffusion and use show that RBK is not sufficient to meet the needs of professionals using it. 
Integrating contextual, local, and practitioner knowledge with research knowledge is critical to 
developing "useable" knowledge to guide action (Cohen & Weiss, 1 993; Honig & Coburn, 2008; 
Hood, 2002; Huberman, 1 990; Lindblom & Cohen, 1 979; Weiss, Murphy-Graham, Petrosino, & 


1 Davies and Nutley (2008) further define RBK as information that includes these research findings, but 

also "research evidence," which they define as findings used in support of a position. We categorize findings used 
in support of a position as a type of knowledge use, rather than a type or form of knowledge. 
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Figure 1.1 Conceptual Framework 
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Gandhi, 2008). Therefore we considered SEA use of other EBK, which we define as data, facts, 
and other information relevant to the problem of school improvement, such as state test scores, 
the distribution of qualified teachers, and formative feedback loops on implementation, and 
practitioner knowledge, which is the information, beliefs, and understanding of context that 
practitioners acquire through experience. 

Organizational structures and social networks 

The formal organizational structure of most SEAs has long been criticized for its hierarchical 
and segmented nature, with little communication across different units and programs. These 
structures are in part a legacy of their institutional history and context, and particularly of federal 
law and programs. Federal categorical funding streams and related regulations spurred the 
growth of distinct divisions and competing "silos" within SEAs and isolated these categorical 
programs from the general education programs administered with state funds (McDonnell & 
McLaughlin, 1982). While some SEAs have reported creating more collaborative structures 
in order to assist low-performing schools under NCLB or state policies (CEP, 2007), many still 
perceive SEAs as largely siloed organizations (Unger, Lane, Cutler, Lee, Whitney, Arruda, & Silva, 
2008). 

In her research comparing two SEAs, Lusi (1 997) argued that non-hiera rchical, less segmented 
SEA management structures could help build internal and external connections and produce 
the kind of adaptive organization that was more effective for complex school reform. While 
sociologists have studied these kinds of connections, known as "social networks," for years to 
understand the diffusion of knowledge and innovation within and across organizations, not until 
more recently have scholars applied a social network perspective in education settings-primarily 
schools and school districts (Daley, 201 0). A few researchers have used social network theory 
and methods to study state education policy networks, most notably Miskel and his collegues 
(Miskel & Song, 2004; Song & Miskel, 2005) who used these methods to describe cohesiveness 
and identify the most central, influential or "prestigious" actors in state reading policy networks. 
But we know of no studies of SEAs that use these lenses to explore the exchange of knowledge, 
and the expertise staff search for, acquire and use in their school improvement strategies. 
We applied them here to study SEAs communication structures and search and incorporation 
networks, and to identify the most central knowledge brokers or influential knowledge sources. 
Drawing on social network literature (see for example, Coburn & Russell, 2008; Frank, Zhao, & 
Borman, 2004; Penuel, Riel, Krause, & Frank, 2009; Spillane, Camburn, Pustejovkdy, Pareja, & 
Lewis, 2008), we also explored factors, such as relational trust and collective responsibility, that 
influenced the nature and quality of the SEA's social networks. 
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Search and incorporation processes 

Though search and incorporation processes are not necessarily distinct in practice, we separated 
them here for conceptual purposes. As we have discussed, SEAs may search for or receive 
information on school improvement both internally from colleagues and externally through 
intermediary organizations. We broadened the idea of organizations' search for knowledge 
and advice to include the nature of the research user-research source interaction, ranging from 
knowledge that is "pulled in" to knowledge that is "pushed in" or "exchanged" (Davies & Nutley, 
2008). RBK in different forms, for example, can be pushed in to the SEA user community from 
intermediaries, or pushed in internally from one SEA department to another, as well as sought 
out (pulled in) or exchanged over time. 

Some researchers have conceptualized search, incorporation, and use of RBK as a social process 
that involves social sense-making or interpretation of varied types of knowledge (see, for 
example, Honig & Coburn, 2008; Spillane, Reiser, & Reimer, 2002; Weick, 1995). It includes 
different forms of RBK, practitioners' knowledge and beliefs, varied expertise, and other types 
of EBK. This incorporation process helps individuals and organizations integrate often de- 
contextualized research findings into their policies and practices (Argyris & Schon, 1996; Honig 
& Coburn, 2008; Honig & Hatch, 2004; Kennedy, 1982). The process can also be generative, as 
learning, working, and innovating can interact such that communities of practice or professional 
knowledge networks actually construct new forms of "useable" knowledge for guiding action 
(Brown & Duguid, 1 991 ; Barnes, Camburn, Sanders, & Sebastian, 201 0). Therefore, we applied 
this lens to the processes that particular networks in the study sites used to search for, obtain, 
and incorporate knowledge into their practices, programs, or guidance. 

Political, contextual, and institutional factors 

Knowledge utilization, policy, and evaluation literatures suggest that whether and how RBK is 
used will be influenced by numerous contextual and institutional factors inside and external 
to the SEA organization. The policy context, such as mandates and incentives requiring or 
encouraging action in a particular sphere, and the regulations that govern action, can shape 
or constrain knowledge search and incorporation. For example, the NCLB requirement that 
states create a statewide system of support for low-performing schools with particular elements 
stimulated searches for data and instructional coaching models among several Midwestern 
states ( Massed etal., 2010). Moreover, the political agenda for school improvement frequently 
shifts. The Race to the Top competition and changes in the Title I School Improvement Grants 
program created funding incentives for low-performing schools to adopt one of four federally 
defined "turn around" strategies. Other contextual factors, such as the supply, numbers, and 
background of SEA staff, and other resources, have also been shown to constrain the types of 
school improvement strategies and technical assistance that SEAs consider feasible (see for 
example, Barnes, Vanover, Salloum, Perrault, Massed, & Rowan, 2011; Massed et al., 2010). 


8 CONSORTIUM FOR POLICY RESEARCH IN EDUCATION | Cpre.org 


Introduction 


These findings about the role of context are consistent with earlier knowledge use studies in 
other settings (see for example, Spillane & Thompson, 1 997; Weiss et al., 2008). Therefore, we 
considered whether each SEA's policy and political context, as well as level of agency expertise, 
staffing and funding, influenced the search, incorporation, and use of research and other types 
of evidence in their school improvement strategies. 

Knowledge use 

Finally, we investigated whether and how RBK and other forms of knowledge were used in SEA 
school improvement strategies, policies, and practices, and the strength of knowledge use that 
emerged as a result of SEA's search and incorporation processes. The strength of knowledge use 
contains several of the dimensions discussed more fully above. It includes, among other things, 
the extent to which multiple forms of knowledge (RBK, especially RBK designed to be used in 
practice, other EBK, and practitioner knowledge) occupy a central conceptual or instrumental 
place in state processes and strategies for school improvement, as reified in documents, tools, 
protocols, processes, or as reported in the perspective of respondents involved in the work. 
As noted earlier, research suggests that the use of multiple forms of knowledge is critical in 
contextualizing and elaborating on research for action. So, for instance, we considered the 
extent to which state strategies sought to include contextual, situational, practitioner knowledge 
as a complement to reliable RBK and EBK, such as by building in feedback loops to formatively 
shape the states' improvement strategies over time, or encouraging districts and schools to use 
these kinds of EBK in their practices in a continual improvement cycle. Research also suggests 
the importance of broad searches on common problems that include a range of expertise, 
and that result in a more coherent vision and approach to school improvement. We, therefore, 
considered as well the extent to which state actors shared a common vision and embedded a 
common, coherent approach to RBK across the school improvement work. 

Study Methods 

The study included a purposive sample of three SEAs that are located in different regions of 
the country and vary in size (from 250 to 500 staff) and organizational structure. The states also 
differed on the extent to which their accountability systems and school improvement strategies 
were focused primarily on schools versus school districts, on the provision of direct assistance 
versus more general capacity-building support, and on the involvement of regional education 
agencies in educational programming and technical assistance. Data for the study were collected 
between late 2010 and summer 2012. Two rounds of in-depth interviews were conducted 
with high-level SEA staff involved directly in school improvement and related programs (e.g., 
curriculum and instruction, accountability, special programs, teacher policy), and with a small 
number of leaders of external organizations that were central to research use in the SEAs. A 
web-based survey was administered to SEA staff in each state and documents describing SEA 
school improvement policies and tools designed for district and school use were collected and 
analyzed. 
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Interviews 

The first round of interviews was conducted on-site in late 2010 and early 2011 with a total 
of 49 senior SEA staff members across the three states. Protocols were created for three tiers 
of respondents for this round. Tier 1 protocols included staff and leadership most directly 
involved in the development, implementation, and/or revision of state strategies for improving 
low-performing schools and school districts. Tier 2 protocols were administered to staff and 
leadership who worked on school improvement issues for low-performing schools and districts, 
but less directly than those inTier 1 . Finally, Tier 3 protocols were designed for staff and leadership 
who were in areas not directly related to school improvement issues for low-performing schools 
and districts, such as assessment, accountability, and teacher policy. 

The content of the interview questions for each tier was based on the study's conceptual 
framework and research questions. Each protocol began with questions aboutthe respondent's 
professional background and the organizational structure of the SEA. The protocol for the Tier 1 
respondents included cognitive interviews that prompted the respondent to outline the state's 
strategy for improving low-performing schools and districts, followed by questions regarding 
the search and incorporation process related to the three different types of knowledge (RBK, 
EBK, practitioner) identified in the conceptual framework. Included in this set of questions were 
prompts regarding the respondent's most important sources of knowledge and how decisions 
were made whether or not to incorporate new knowledge into the SEA's strategy/strategies for 
improving low-performing schools and districts. The protocol for Tier 2 respondents addressed 
these same topics, but at a more general level. The protocol for Tier 3 respondents focused 
on the organization of the SEA and any involvement the respondent had in the state's school 
improvement strategies.TheTier 3 protocol also included modules containing specific questions 
aimed at respondents with specialized expertise in areas such as curriculum, accountability, and 
Title I. 

A second round of interviews was conducted in summer 201 2 with 1 1 senior SEA staff across 
the three states who, based on our analysis of the Round 1 interviews and the survey data, were 
centrally involved in knowledge networks for improving low-performing schools and districts, 
and with two staff from intermediaries who emerged through interviews and the survey 
as particularly influential sources of RBK for these SEA actors. The protocol for the SEA staff 
focused on changes to the SEA's structure and school improvement strategies, and gathered 
more in-depth information on the respondents' communities of practice. The protocol for the 
external providers gathered information on the provider's services, relationship and process of 
working with the SEA, and the provider's use of research and other types of information in the 
development of its products and services for school improvement. 

All interviews were recorded and transcribed for analysis. Codes for analysis of the Round 1 
interviews were initially developed based on the study's conceptual framework, research 


10 


CONSORTIUM FOR POLICY RESEARCH IN EDUCATION | Cpre.org 


Introduction 


questions, and analysis plan, and new definitions were created as needed. Using the online 
qualitative data management tool Dedoose,each member of the research team coded the same 
interview in order to discuss the applicability of the codes and their definitions. Based on this 
process, the code list and definitions were then revised. To ensure inter-rater reliability, the entire 
team initially coded the same sections of one interview from each of the three states (three tests) 
and calculated the Pooled Cohen's Kappa for each test. The team discussed commonalities and 
discrepancies in coding and refined the codes and definitions after each of the tests. 2 Each team 
then used Dedoose to code the interviews for the SEAs in which it had conducted the site visits. 
(See Appendix Table 1.1 fora list of the major codes.) 

Drawing on interviews and documents, each team prepared an internal background paper that 
described each SEA's school improvement policies and processes and the organizational and 
institutional context of these activities. These papers were used to create the profiles in Chapter 
2. Interview data and document reviews also provided detail on the sources, types, and qualities 
of research or other forms of knowledge that SEA staff sought and used in school improvement 
decision-making and the knowledge search and incorporation process. For these and other 
analyses, the lead author(s) of each report chapter compiled and analyzed coded interview data 
across the three states in their topic areas. 

Survey 

A web-based survey was administered to SEA staff in the three states in late 2011 and early 
2012. The survey was based on the study's conceptual framework and research questions, 
and was developed simultaneously with the interview protocol to ensure that it covered similar 
content (although at a different level of specificity). Preliminary feedback from the interviews of 
SEA staff was also used to inform the final survey instrument. 

The survey collected information on: 1) the respondents' length of service in the SEA; 2) the 
use of different forms of knowledge in their work; 3) the organizational units and individuals 
inside and outside the SEA from whom they sought advice and information in four networks: a) 
general work network; and b) research, c) other evidence/data, and d) practitioner knowledge 
networks related specifically to programs and practices for low-performing schools and school 
districts; 4) the strength of those networks (as measured by frequency of communication and 
their influence on respondents' work); 5) the level of trust and collective efficacy within each of 
the networks; and, 6) other influences on respondents' work. 

Survey sample and response rates 

A total of 610 surveys were sent to SEA staff across the three states. The survey sample differed 
across the states, based on the size of the SEA. All professional staff were surveyed in State 

2 The Pooled Cohen's Kappa on the final test ranged from 0.73 to 0.82 with an average of 0.77. Scores 

ranging from 0.6 to 0.8 are considered acceptable, while scores above 0.8 are considered very strong. 
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C which had the smallest SEA (194 surveys). In State B, all professional staff outside the 
Department of Administration, and a large sample of staff in that department, received a survey 
(total of 245 surveys). Because of the large size of its SEA, surveys were sent to all professional 
staff only in the departments of school improvement, curriculum and instruction, accountability, 
assessment, and research in State A. Staff in the other offices and departments were sampled, 
but included all office directors, assistant and associate commissioners, and the commissioner 
of education (total of 171 surveys). Two reminder emails were sent to non-responders in order 
to increase completion rates. The overall response rate was 73.8%, but ranged from 64.9% in 
State Ato 83.5% in State C (Appendix Table 1.2). 

All 450 survey respondents were asked whether their work related "in any way to improving 
low-performing schools and school districts" in their state. Only those who answered "yes" to 
this question, that is, staff who self-identified as being involved in school improvement work 
regardless of the SEA office in which they worked, were asked questions about their research, 
other evidence/data, and practitioner knowledge networks related specifically to programs and 
practices for low-performing schools and school districts, trust and efficacy within these networks, 
and other influences on their work. Because of the study's focus on the use of evidence in school 
improvement policy and practice, the findings included in this report are based on answers from 
the 305 survey respondents who self-identified as working in this area. 3 

Network questions 

For the general work network, respondents were asked "With which SEA offices or external 
organizations do you interact when discussing issues related to your work responsibilities?" 
For each of the three knowledge networks, respondents were asked: "To which SEA offices or 
external organizations do you turn when seeking [research, data, advice from practitioners] on 
programs and practices targeted at improving low-performing schools and school districts?" 
To keep the network questions conceptually distinct, the survey provided definitions of the 
terms "research," "data," and "practitioner" that were comparable to those used in the study's 
conceptual framework and interview guides. 

Respondents were asked five sets of questions for each knowledge network. They were first 
asked to select, from a prescribed list, all of the SEAs offices or departments from which they 
sought information or advice, and to write in up to five external organizations or groups that 
were also sources of information. They were also given the option to reply: "I do not seek 
[research, data, practitioner advice] from any SEA office about school improvement in order to 
fulfill my work responsibilities." Second, respondents could list up to three individuals "from 
whom you seek [research, data, practitioner advice] about school improvement" in each SEA 
office/department and external organization they had identified. Third, respondents were asked 


3 Data was collected from all survey respondents on their general work networks and general research 

networks. Time and resource limitations precluded analysis of these data. 


12 


CONSORTIUM FOR POLICY RESEARCH IN EDUCATION | Cpre.org 


Introduction 


how often they had interacted "in this school year" with each office/department, organization 
and individual they named: a few times a year, once or twice a month, once or twice a year, 
or daily, or almost daily. Fourth, they were asked how influential the advice from each person 
or organization was on their work, using a 4-point scale that ranged from "not influential" to 
"highly influential." Finally, respondents were asked a set of trust and efficacy questions about 
the group of individuals they had named in the preceding questions. (See Appendix Table 1.3 
for the trust and efficacy questions.) Respondents were asked similar types of questions for the 
general work network. 

Due to concerns raised by State A, questions asking respondents to name individuals, and all 
trust and efficacy questions were removed from its survey. Only the names, and frequency and 
influence of offices, departments or external organizations, were collected. 

The network survey questions were derived from prior network surveys, including items used 
in the Weinbaum, Cole, Weiss, and Supovitz (2008) study of teacher communication networks 
around instructional reform in high schools. Similar question types, which asked respondents 
for names and closeness of contact with peers or other organizations, have been used in previous 
research that included social network survey and analysis (see for example, Burt, 1 992; Miskel, 
Coggshall, DeYoung, Osguthorpe, Song, & Young, 2003; Podolny & Barron, 1997). Trust and 
collective efficacy measures were adapted from Bryk and Schneider (2002); Goddard (2001); 
Goddard, Floy, and Woolfolk Floy (2000); Goddard and LoGerfo (2007); and, Goddard, Salloum, 
and Berebitsky (2009). 4 

Survey analyses 

The survey data were used to identify internal and external sources of research and other types 
of knowledge, and to analyze the size, strength, and configuration of the networks, including 
patterns of cross-office or within-office communication. We identified the most central, highly 
connected information brokers and influential individuals, offices or organizations in the 
networks using a rank order of standardized centrality measures, along with sociograms. We 
drew on the direct, interpersonal networks of these influential individuals, along with interview 
data, to corroborate and interpret the broader network analyses, and to provide more specific 
information on the internal and external sources of research or other knowledge. 

The configurations of the general work network and three knowledge networks were mapped 
using UCInet adjacent matrices and NetDraw, a network visualization software program 
(Borgatti, Everett, & Freeman, 2002). (Chapter 3 provides greater detail about this process 


4 The trust and efficacy survey questions used in this study were analyzed by Goddard prior to the survey 
being administered. His analysis suggested that two negatively worded collective efficacy questions be reworded, 
as well as one trust item. These changes were made prior to the survey being administered. Overall, Goddard's 
results suggested adequate factor structure and strong reliability for the subscales. 
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and the calculation of the measures mentioned below.) Sociograms were created that showed 
connections, and the strength of these connections, among SEA staff for all four networks and 
between SEA staff and external organizations for the three knowledge networks. Connections 
among SEA members are two-way ties because it is possible for an individual or office to be the 
originator (in this case, the seeker of research information) or recipient (in this case, the named 
source of research information) of ties. The former is considered an "out-tie," while the latter is 
considered an "in-tie." One way we examined networks was to use a count of in-ties and out- 
ties, or both, within and across formal SEA departments.To examine ties between SEA members 
and external members of networks, we used out-ties from our SEA respondents that formed 
in-ties to externals as we interviewed a subsample, but did not survey, external intermediaries. 
We also used UCInet to analyze the full networks' size, inclusiveness, density, and degree of 
centralization (degree centralization and betweenness centralization). 

Strength of network connections was measured though a combination of respondents' reports 
of the frequency of their communication about research or other kinds of information, and the 
influence they perceived the resulting information to have on their work. Degree of strength 
was considered using a matrix ranging from a cell defined by "highly influential/daily contact 
(200)" to a cell defined by "not influential/a few times per year contact (0.5)." (See Appendix 
Table 1.4.) The scales for measures of frequency of communication and degree of influence 
were drawn from Weinbaum et al. (2008). 

The relative centrality of individual actors or offices within the networks was analyzed using 
UCInet and qualitative data, more specifically with Freeman'sdegreecentralityand betweenness 
centrality (Scott, 2013; Song & Miskel, 2005). The study data are "directional" because we have 
an interest in not simply that a connection between actors exists, but also in the direction of their 
ties. Thus, we use "in-degree" as well as "out-degree" centrality (Scott, 201 3). The sociograms 
were also used to identify the more influential external intermediaries, and to understand 
network configurations. 

Documents 

The research team identified and collected hard copy and web-based documents from each 
state. These included the SEA's formal organization charts, school improvement frameworks, 
descriptions of school improvement policies and processes, tools developed for use by school 
districts and schools and technical assistance providers, and, if available, evaluations of a state's 
school improvement programs. The organization charts were used to identify key interview 
sources and survey respondents and to structure analyses of formal and informal communication 
networks. The other documents were used to describe school improvement policies and the 
types of support SEAs provided to low-performing schools and school districts. The documents 
were also searched for citations to research and other kinds of knowledge. 
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Overview of the Report 

The remainder of the report presents findings about how the three SEAs in our study: 1) 
searched for research-based, evidence-based, and practitioner knowledge related to school 
improvement; 2) were organized to manage and use such knowledge; and, 3) used research 
and these other types of knowledge to design, implement, and refine state school improvement 
policies, programs and practices. Chapter 2 profiles the organizational structure and capacity 
of each SEA, the legal, fiscal, and political environments in which it operated, and its school 
improvement strategies and delivery system to provide a context for the analyses reported in 
subsequent chapters. Chapter 3 examines the strength, structure, and flow of information in 
the SEAs work and knowledge networks, and identifies the most influential and well-connected 
individuals, offices, and/or external organizations. Chapter 4 delves more deeply into these 
social networks, analyzing the direct connections or "ego networks" of the most influential or 
well-connected information brokers, and the role of these core knowledge networks in the SEAs 
search and incorporation processes. Chapter 5 looks more specifically at SEAs relationships 
with external organizations, examining the extent to which, where and why SEA staff turned to 
external intermediaries for research and other types of knowledge about school improvement. 
Chapter 6 focuses on the types and forms of research and other kinds of evidence that SEAs used 
in their school improvement policy and the factors that appeared to explain this use. Chapter 7 
summarizes the study's findings and discusses implications for policy and research. 
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CHAPTER 2 


SEA Context, Organization, and 
School Improvement Strategies 


In the conceptual framework presented in Chapter 1, we identified a set of institutional and 
contextual factors inside and external to SEAs that can influence how SEAs seek, manage, and use 
research-based (RBK) and other forms of knowledge. These include the organizational structure 
and capacity of SEAs, and the legal, fiscal, and political contexts in which they operate. This 
chapter provides brief summaries of these factors for each of the three study states as context 
for the analyses reported in subsequent chapters. As the study focuses on the search for and 
use of knowledge in school improvement policies and practices, the profiles also describe each 
state's school improvement strategies and delivery systems. The chapter concludes with a brief 
discussion of differences and similarities across the three states. 


State A's school improvement and accountability policies build on nearly 20 years of state 
legislation and regulation and are now deeply intertwined. Most recently, legislation and 
State Board of Education regulations enacted in 2010 established a legal framework for 
identifying and providing targeted support to students, schools, and school districts most in 
need of assistance and provided new authority for districts and the state to intervene in the 
lowest-performing schools and districts. The law and regulations, developed in consultation 
with a legislatively mandated advisory council, created: 1) a five-level (Stages 1 to 5) system 
of differentiated district and school accountability and assistance; 2) district standards, and 
indicators and conditions for school effectiveness to be included in the district standards; 3) 


and Delivery Systems 
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Context 

Legal context 
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criteria for placing schools and districts into each level of accountability and assistance; 4) 
requirements for district improvement plans and school turnaround plans; and, 5) processes for 
moving schools and districts out of the lowest levels of accountability. The state has integrated 
federal accountability requirements into its own system. 

Political context 

State A has had the same governor and commissioner of education since 2008. Both are strong 
proponents of education reform as a means of improving education and closing the achievement 
gap in the state. The governor released an education agenda early in his administration 
that focused on raising the achievement of all students, including addressing out-of-school 
circumstances, professionalizing teaching, broadening and deepening commitment to public 
education, and fostering innovation. The commissioner of education, working with the State 
Board of Education, subsequently identified four priorities, aligned with those of the governor, 
to guide the work of the SEA: 1) effective educator policies; 2) improvement of curriculum and 
instruction; 3) integration of accountability and assistance efforts; and, 4) incorporation of 
student and family supports. 

Fiscal context 

Like most states, State A was affected by downturns in the national economy. The overall state 
budget remained flat between 2008-09 and 2011-12 through a combination of budget cuts, 
increased federal aid, and state tax increases. Spending on K-12 education decreased about 
8% during this period. The budget for SEA operations dropped 7% and was then flat funded 
between 2009-10 and 201 1 -1 2. 

Organizational Structure and Capacity 

Organizational structure 

At the time of our initial round of interviews and administration of the SEA survey, the SEA 
had nine departments reporting to two deputy commissioners, and a tenth department, 
Administration, reporting directly to the commissioner. (See Figure 2.1 .)The Division of Learning 
and Data included the departments of Research and Curriculum and Instruction, and the offices 
of Assessment and Data Analysis and Reporting (which are located in separate departments). 
The Division of School Improvement and Assistance included the departments of Accountability, 
School Improvement, Student Support, Educator Preparation, and Special Education. 1 When 


1 The SEA was reorganized slightly in September 201 1 and then again in August 201 2. The Department of 

Administration now reports to a chief operating officer. The Department of Accountability now reports to the Direc- 
tor of School Improvement, and the offices of Vocational and Career Education were moved into the same division 
as curriculum and instruction. 


1 8 CONSORTIUM FOR POLICY RESEARCH IN EDUCATION | Cpre.org 


Figure 2.1: State A 


SEA Context, Organization, and School Improvement Strategies and Delivery Systems 



19 






















State Education Agencies' Acquisition and Use of Research Knowledge in School Improvement Strategies 


asked about the rationale for this organization, a senior SEA respondent replied that it was done 
so that each deputy commissioner would supervise roughly the same number of staff. 

The primary responsibility for implementing State A's accountability and school improvement 
policies resides jointly in the departments of School Improvement and Accountability. State 
law requires that the SEA have separate units overseeing accountability and assistance, with 
both reporting to the same administrator. This structure was designed to send a signal to school 
districts that while accountability and assistance are discrete processes, they are intended to 
support each other. As several respondents explained, the SEA views "accountability as the 
highest form of assistance." That is, monitoring reports no longer just identify problems, but 
are used to trigger and inform supports to schools and districts. The two departments work 
closely together. As a director in the School Improvement Department noted: "The director of 
Accountability and I sort of are the yin and yang of the [assistance and accountability] work." 

The small (4 person) Department of Accountability is responsible (with support from the 
Research Department) for identifying schools and districts under the state accountability system, 
conducting formal reviews of all Stage 4 districts (containing one or more of the state's 4% lowest- 
performing schools, around 9 to 10 local education agencies [LEAs]), and monitoring progress 
of Stage 4 district improvement plans. Reviews are conducted by teams of generally four to 
six consultants that include a former superintendent, a director of curriculum and instruction 
or principal with a strong background in that area, and a former financial officer. All had had 
a variety of district- and school-level roles. The Department of Accountability also conducts 
reviews of Stage 3 districts (containing one or more schools among the lowest-performing 20% 
of schools but no Stage 4 schools, around 70 LEAs), although budget constraints have limited the 
number of Stage 3 district reviews that the SEA can actually do. As a result, the SEA had prioritized 
these districts for review, focusing on those districts that have the lowest achievement, are not 
showing growth and/or have leadership and governance challenges. Both Stage 3 and Stage 4 
reviews contain recommendations that provide a basis for the design and delivery of technical 
assistance, reflecting the SEA's goal of making the reviews transparent and of use to the districts, 
rather than creating large documents that will sit on administrators' shelves. 

The Department of Accountability also conducts specialized studies focused on narrowing the 
achievement gap. They are designed to identify district and school factors contributing to 
improvement in achievement for students with specialized needs (students with disabilities in 
2009, English language learners (ELL) in 2010, and students living in poverty in 2011) in selected 
schools in order to identify and promote the dissemination of promising practices. The studies 
are conducted in districts with schools that have succeeded in narrowing the achievement gap 
for the subgroup under study that year, and use data from school and district monitoring reports 
and school grantapplications. 

The Department of School Improvement is the assistance arm of the SEA. It is charged 
with providing targeted assistance to districts identified as in need of improvement and for 
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districts that have schools in need of improvement (primarily Stage 3 and 4 school districts). 
As described below in the section, School Improvement Strategies and Delivery System, an 
urban assistance unit provides customized support for the largest urban districts in the state, 
nearly all of which are Stage 4 districts. A regional assistance unit provides support for all 
other districts, primarily through six Regional Assistance Teams that were created in late 2009. 
Priority is given to Stage 4 districts not served by the urban assistance unit and to many of the 
Stage 3 districts. Stage 1 and 2 districts may participate in regional networks and, to the extent 
permitted by Regional Assistance Teams resources, other Regional Assistance Team activities. 
Regional Assistance Teams are virtual; they do not have permanent offices and sometimes 
rent space from area higher education or other education-related institutions. 2 Each Regional 
Assistance Team is composed of a part-time director (who is a retired superintendent), a data 
specialist, a mathematics specialist (with knowledge of special education), a literacy specialist, 
a professional development coordinator, and a support facilitator (often a retired principal) who 
work with schools to implement SEA school improvement tools. Most of the Regional Assistance 
Team members are consultants who are hired through, supervised, and supported by external 
organizations through contracts with the SEA. 

One SEA respondent reported that a prior commissioner had "blown up" the Title I office 
in an attempt to break down "fiefdoms" within the SEA. A School Improvement Grants (SIG) 
program office, located in the Department of School Improvement, identifies schools that fail 
to meet federal accountability criteria, and administers the Title I program, including the Title I 
SIG program. As described above, however, other school improvement activities are housed in 
other offices in the Department of School Improvement and the department's director uses SEA 
Title I funds to support related activities in his office and in other parts of the agency, such as the 
Department of Curriculum and Instruction. 

The Department of Curriculum and Instruction was created in 2008; prior to that time, 
responsibility for curriculum was spread throughout the agency. It has separate units for literacy/ 
humanities, math/science, ELL, and instructional technology. As described in a later section, 
specialists from Curriculum and Instruction work closely with staff of the Department of School 
Improvement, serving as content network coordinators and as members of the state's Regional 
Assistance Teams. 

Responsibilities for the collection, analysis, and reporting of data are spread throughout the 
agency. The Assessment Office, which is responsible for coordinating the development, 
administration, and reporting of the state's statewide student assessment programs, is located 
in a different department from the Data Analysis and Reporting Office, which collects student, 
educator, and school data, develops statistical reports and school and district profiles, and 


2 State A abolished its regional offices a number of years ago. The SEA respondents were clear that the 

Regional Assistance Teams are not meant to replace these offices which had served as general assistance providers 
and monitors. 
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oversees and responds to data inquiries from inside and outside the SEA. Staff in the Research 
Department compile and prepare data reports for the Accountability Department. The SEA has 
an agency-wide data council composed of the managers of the units that do analysis work that 
meets regularly to coordinate and align their work. 

The primary mechanism for cross-agency communication is regular senior staff meetings 
(department heads and deputy commissioners) held with the commissioner. Towards the end of 
this study, the SEA implemented a new performance management system organized around the 
agency's major goals (e.g., school and district turnaround, educator evaluation, strengthening 
curriculum, instruction and assessment). Cross-agency teams with responsibilities related to 
a goal develop strategic plans, identify priority projects, and set and monitor progress toward 
goals, reporting on a regular basis to senior staff and the commissioner. The SEA also created 
ad hoc cross-agency teams to design specific programs. For example, the director of school 
improvement led an internal task force to design the state's response to intervention program 
that included mid-level staff from curriculum and instruction, special education and social- 
emotional health offices. 

The SEA also receives input from standing and ad hoc advisory committees. For example, a 
legislatively mandated Advisory Council on School and District Accountability and Assistance 
advises the State Board of Education on the development and implementation of the state's 
accountability and assistance system and the policies and practices of the Departments of School 
Improvement and Accountability. Its members include representatives of the state's major 
education associations and business, and researchers from local universities. The state also 
creates ad hoc advisory committees as specific policy needs arise, such as a taskforce composed 
of educators, representatives of education associations and researchers that recommended 
a framework for evaluating teachers and administrators in the state. Subsequent legislation 
created an educator evaluation data advisory committee, composed of SEA staff, legislators, and 
stakeholders, to identify data necessary to assess the effectiveness of district evaluation systems. 

SEA capacity 

State A's SEA had around 500 employees in 2011. About 35% of these staff focused on SEA 
administration, information technology and program compliance. Unlike other SEAs, State A did 
not suffer major cuts in staff during the recession, although vacant positions were left unfilled 
and some departments, like Accountability, are viewed as under-staffed. The Department of 
Curriculum and Instruction merged two of its offices, literacy and humanities, when one of the 
directors left as a way of managing scarce resources. One strategy for managing work within 
existing resources has been to bound the agency's responsibilities. For example, state law limits 
the number of Stage 4 schools to 4% of the state's schools and the number of Stage 5 districts 
(those under joint state-district governance) to 2.5% of the state's LEAs. As noted above, budget 
constraints (funding for these reviews was cut by one-third) limited the number of Stage 3 
district reviews that the SEA conducts. 
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The directors of the major departments were a veteran and stable group. While the two 
deputy commissioners retired during the course of our study, one after more than 30 years 
in the SEA, the directors of the Accountability and School Improvement Departments had 
worked in the SEA between 15 and 30 years. The directors of the Department of Curriculum 
and Instruction and Research had held their positions since their departments were created in 
2008 and 2007, respectively. Staff in offices within the School Improvement Department also 
had many years of experience in the SEA, often after careers in local school districts. SEA staff 
held their colleagues in high regard, often citing them as sources of assistance and information. 
Research staff have research backgrounds. The School Improvement Department staff (and 
their deputy commissioner), on the other hand, came to the SEA from positions as educators 
and administrators in local school districts. This reflects a desire by these staff and the current 
commissioner to "look outward" to their clients. 

Research capacity in the SEA 

State A's SEA has considerable research capacity. The staff members most responsible for 
producing and disseminating RBK throughout the SEA are located in Department of Research, 
an office with around 1 2 to 1 6 staff. The current iteration of this office dates to March 2007. The 
previous research unit was disbanded at the start of the standards and assessment movement to 
move its staff into the assessment unit. The Department of Research has multiple responsibilities: 
1) respond to research-related inquiries from other parts of the SEA (e.g., research reviews); 2) 
conduct in-house research; 3) develop analytical tools for the SEA and the field; 4) evaluate state 
and federal programs; 5) prepare data reports for and brief District Review teams in the Office of 
Accountability; 6) manage and provide implementation support for the state's Race to the Top 
grant; 7) support other SEA offices and the SEA's strategic planning process; and, 8) serve as a 
clearinghouse (basically on the web) for all research, evaluation, and data products released by 
the SEA. Some of this work is done in-house, and some (particularly program evaluations and 
literature reviews) is contracted out. 

School Improvement Strategies and 
Delivery System 

At the center of State A's accountability and assistance actions is a framework that establishes five 
levels of differentiated accountability/assistance. Three key principles guided the development 
of this framework and provide the theory of action underlying the State's school improvement 
policy and actions. First, the district is the entry point for the SEA's accountability and assistance 
work. This district approach was taken in recognition that: 1 ) the SEA does not have the capacity 
to work directly with a large number of low-performing schools, and 2) schools that had received 
intensive state support often back-slide when that support is removed if there is insufficient 
district capacity to maintain improvement efforts. The focus of state assistance is on building 
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district capacity to support and guide improvement efforts in individual schools; therefore, a 
district's placement in one of the state's five accountability stages is determined primarily by 
the designation of its lowest-performing school. The SEA does not provide direct support to low- 
performing schools unless required by external funding (e.g., Title I SIG, Reading First). 

Second, a strong accountability system will not, by itself, result in continued improvement. 
A parallel system of assistance and intervention is necessary to secure continued, strong 
improvement. Third, levels of accountability and intensity of assistance and intervention need to 
match the severity and duration of any identified problems. The framework includes a set of six 
district and 1 1 school standards, and related indicators, that form the core of district and school 
accountability, planning and assistance at all levels of the system-SEA, district and school. The six 
district standards focus on leadership and governance, curriculum and instruction, assessment, 
human resources and professional development, student support, and financial management. 
Indicators of effective practice at both the school and district levels are embedded in each district 
standard. The district standards date to the late 1 990s and were updated a decade later after a 
review of extant research and input from stakeholder groups. 

The Department of School Improvement, along with other SEA offices, provides a range of 
differentiated assistance to school districts. This includes: 1) customized support for the 10 
largest urban districts (which account for over 80% of the state's lowest-performing schools); 
2) a regional system of support for all other districts; 3) district and content networks; and, 4) 
web-based tools designed for use by all districts. The urban assistance unit provides customized 
support primarily through two activities. First, each year the unit develops a Memorandum 
of Understanding with each district outlining the resources and assistance to be provided to 
strengthen the district's capacity to lead and monitor school improvement. These activities 
might be supported by funds provided through the SEA's grant process. Second, a member of 
the urban assistance unit is assigned to each district as a liaison. Their responsibilities include 
serving as an SEA point of contact for the district, brokering SEA and external resources, and 
assisting districts in the development and implementation of district and school improvement 
plans (SIPs) and state and federal grants. 

As described in an earlier section, Regional Assistance Teams support other school districts. 
These teams support district self-assessment and planning, facilitate planning for and access to 
professional development, 3 train school and district staff to implementSEA resourcesand school 
improvement tools, and serve as a forum for regional networks of school and district teams on 
various topics, such as the education of ELL students or students with disabilities. The Office 
of School Improvement supports a network for urban superintendents and the Department 
of Curriculum and Instruction (in collaboration with the School Improvement Office) supports 


3 The Regional Assistance Teams also provide a limited amount of professional development in leadership, 
core curriculum standards, literacy, mathematics, sheltered inclusion for ELLs and special education/inclusive 
practice to districts and schools through summer courses and follow-up sessions. 
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content networks in mathematics, literacy, and English language learning. The content networks 
bring together SEA content and district curriculum specialists on a regular basis to share ideas, 
resources, and strategies for improving teaching and learning. 

Finally, the SEA has developed a number of school improvement tools that are available to all 
districts on its website. Several of these tools were developed in collaboration with the urban 
superintendents' network and their districts in response to commonly identified needs.The SEA 
also turned to external organizations to support the tool development. Tools enable schools and 
districts to conduct data analyses and data-driven decision-making within a cycle of inquiry, pro- 
vide rubrics for assessing current practice in each of the school standards, and guide educators 
in implementing professional learning communities and learning walkthroughs. 

Funding of the School Improvement 
System 

The major source of funding for the SEA's school improvement activities was the federal 
government, primarily through Title I. In addition to the Title I Part A program, the SEA awarded 
Title I School Improvement Grants to 35 schools. State A also provided nearly $7 million a 
year in grants to low-performing districts for their use in school improvement activities, such 
as professional development, that were tied to their mandated district improvement plans. 
Participating districts used funds from the state's Race to the Top grant to support school 
turnaround as well as other improvement activities, such as wraparound zones or high school 
reform. In addition to resources focused directly on school improvement, the state had made a 
major financial investment in extended time programs and in services to students who perform 
poorly on the state assessment. 

Funding for the staff and activities of the Department of Research come from multiple sources: 
internal funds, federal grants (such as Race to the Top), program funds, and external grant 
competitions (such as IES at the U.S. Department of Education). 

State B 


Context 

Legal context 

State B has a state accountability system that is used to determine school accreditation status. 
Enacted in 2003, the system includes components for student achievement, measuring both 
status and change, and schools' self-assessment of process indicators aligned to the state's 
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school improvement framework. The SEA calculates and reports state accountability status 
as letter grades. The principal accountability system, however, is that embodied in ESEA. As 
described below, the SEA uses federal criteria to identify and support low-performing schools. 

Political context 

The shape and direction of school improvement policy in State B is driven more by external than 
internal political factors. The state created its statewide system of support (SSOS) in response to 
the NCLB Act of 2001 . More recently, the state enacted new school reform legislation in 2009 
that mirrors language in the federal Race to the Top and Title I SIG programs. These changes 
included the creation of a state school redesign office to oversee the lowest-performing schools, 
and the coordination of components of the SSOS with the requirements of the SIG program. 

Fiscal context 

State B has faced difficult economic times for many years. State revenues declined steadily, 
forcing the former and current governors to cut state agency budgets, including the SEA, to 
address deficits. New federal funding programs, such as American Recovery and Reinvestment 
Act of 2009 and the Title I SIG program, infused new resources into the SEA for assessment and 
school improvement work even as the governor cut state funding. The administrative budget for 
the SEA in 2011-12, however, remained below that for 2008-09. An early retirement program 
facilitated reductions in state agency staffing. 

Organizational Structure and Capacity 

Organizational structure 

State B's SEA is organized into three divisions, each headed by a deputy commissioner who 
reports to the commissioner of education: Administration, Learning and Student Services, and 
Early Childhood Education. An Office of State School Reform reports directly to the commissioner. 
(See Figure 2.2.) 

The Division of Learning and Student Services oversees most of the substantive work related to 
K-1 2 education in the state, and most of the SEA's school improvement activities. It is composed 
of eight departments, including School Improvement, Field Services (Federal Programs/Title I), 
Assessment and Accountability, Educator Preparation, and Special Education. The Department of 
School Improvement is the lead office in the state's school improvement efforts, working closely 
with the Departments of Field Services and Assessment and Accountability. Their structures and 
responsibilities are described below. 

Composed of four offices-School Improvement, Curriculum and Instruction, CharterSchools, and 
Educational Technology-the Department of School Improvement houses most of the SEA's 
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substantive academic and improvement responsibilities. Its director described the department's 
responsibility as: "the improvement of student achievement through our school improvement 
work with high- priority schools or the school improvement grants or just Title I initiatives in 
general regarding school improvement." The Office of School Improvement oversees the 
Statewide System of Support (SSOS) described below. The director of the department felt that 
the inclusion of the Curriculum and Instruction office, as well as the state's adoption of the 
common core state standards, have increased the department's focus on instruction and at risk 
students in State B's high priority orTitle I schools and districts. The department has also started 
a high school initiative to redesign high schools at risk of failing. 

The Department of Field Services was spun off from the Department of School Improvement 
a few years ago, as the director at the time and many others thought that the larger department 
had become too large and difficult to manage. Five field services teams, based in different parts 
of the state, oversee districts' and schools' use of most federal funds from the ESEA (excluding 
special education). Their responsibilities include allocating federal funds, helping schools and 
districts develop grant applications, reviewing and approving these applications, and monitoring 
grant expenditures. As described below, Field Services staff serve on school support teams. ELL 
programs are housed in the Field Services Department as well. 

The Department of Assessment and Accountability is responsible for all statewide K-12 
testing programs, school accountability, and evaluation and research. This department 
develops, administers, and reports the results of all state assessments. Department staff 
calculate Adequate Yearly Progress (AYP) and a state-level school accreditation measure, and 
identify the state's lowest achieving or "priority" schools. The department also provides data, 
data analysis tools, and technical assistance in interpreting and using assessments and data to 
staff inside and outside the SEA. As the director explained: "When people from field services 
or school improvement need help in understanding why a school got where they are, we can 

sometimes help them with that Our on-line Professional Learning System helps people in 

schools understand the appropriate set of tests for their students." The SEA website contains data 
files that allow schools, districts and SEA staff to compare the student performance of a school 
with statewide averages. Department staff also oversee professional development projects for 
school coaches and other teacher leaders on the use of formative assessment data to improve 
instruction. An external advisory committee provides stakeholder feedback and advice to the 
department director. 

Legislation enacted in 2009 in response to the federal Race to the Top grant competition 
requirements created an Office of State School Reform to oversee improvement efforts in 
the state's "priority" schools, the 5% lowest-performing schools in the state. The office, which 
reports directly to the commissioner of education, assists in the development of required SIPs, 
and reviews, approves and supervises implementation of these plans. The Office coordinates 
the provision of support services with the Department of School Improvement, which operates 
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the federal Title I SIG program and statewide system of support forTitle I schools, and with other 
departments in the SEA. Priority schools that fail to make adequate progress on their redesign 
plans may be placed in a special commissioner's district. 

Although not part of the SEA, State B's system of area education agencies (AEA) plays a critical 
role in supporting school districts, charter schools, and the state's school improvement delivery 
system. The AEAs are funded through a combination of stateand local revenues and grants. While 
each AEA responds to the specific needs of its service area, all offer leadership, programs, and 
services in the areas of instruction, career and technical education, special education, technology, 
and finance. As discussed in later chapters of the report, State B's AEAs and their professional 
association have historically partnered with the SEA in the development and implementation of 
education policy in the areas of curriculum, instruction, and school improvement. 

SEA capacity 

At the time of our study, State B had approximately 300 staff in its SEA, a dramatic reduction from 
2000 employees in 1 992. The SEA had recently lost a large number of veteran staff through an 
early retirement initiative and a hiring freeze was in effect at the time of our visits. Respondents 
were divided in their reports of having adequate staff resources for doing the necessary 
work: Some felt they did not have the needed staff and others felt there was enough staff to 
do all of the work. But, overall, the departments involved in school improvement, especially 
the Departments of School Improvement and Assessment and Accountability, remained at 
prior staffing levels and, in some cases, were able to add positions. This was due in part to 
state funding of additional positions in the Department of Assessment and Accountability as 
assessment responsibilities were shifted back into the SEA, and an infusion of Title I SIG funds 
into the Department of School Improvement. 

While top leadership (at the department level) in the SEA was relatively stable, there has been 
some staff turnover. For example, within the last year of this study, the Department of School 
Improvement lost its assistant director, research coordinator, and a consultant, as well as the 
director of Curriculum and Instruction. 

While some respondents felt that the retirement incentives had led to a "brain drain" in the 
SEA, for the most part respondents perceived their staff, leadership or colleagues to have the 
work habits, knowledge and expertise needed to do their jobs. Looking at staff and leadership 
experience or education based on their background accounts confirms this overall impression: 
several staff have Ph.D.s and research and evaluation experience; others have years of 
experience in fields related to school improvement, instruction, and so on. Many of the staff 
we interviewed are former SEA staff or leaders who retired, but were hired back in particular 
offices or divisions-especially related to instruction, assessment and school improvement. As 
one consultant explained: "I have been retired before.They kept me as a contractor. And so I just 
kind of float in and out." 
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Experienced people had also moved from local school districts or AEAs into the SEA as consultants. 
For example, the manager of the SSOS had designed and managed the original School Support 
Team process at low-performing schools and the overall School Support Team network when she 
worked at a key AEA. She developed the tools, processes and reporting forms for this team and 
was generally respected in the field when she was hired by the SEA to manage the statewide 
system. Talent flowed in the other direction as well. A key external contact for the manager of 
the SSOS, for example, who had worked in the Department of Field Services, now works in an 
AEA, bringing the experience and expertise with the ESEA legislation that she developed in the 
SEA to the table as the AEAs and SEA collaborate on the SSOS and school or district compliance 
with federal law. 

Research capacity in the SEA 

The Department of Accountability and Assessment includes an Office of Evaluation and Research 
that, on paper, is responsible for conducting and providing evaluation research to other SEA 
offices, and "overseeing the work of collaborative research partnerships." Research activities 
related to school improvement, however, appear to be decentralized. For example, the 
Department of School Improvement developed an RFPand subsequently oversaw an evaluation 
of the SSOS conducted by a national research organization. And one person in this department 
had been assigned to searching for research on school improvement from such sources as the 
regional comprehensive assistance center, Ed Trust, the national Center on Innovation and 
Improvement, and universities. 

School Improvement Strategies and 
Delivery System 

State B's state accountability and school improvement delivery system is built around a model 
of continuous school improvement (i.e., gather data, analyze/study the data, develop SIP, 
implement, and monitor). The state's School Improvement Framework (SIF), a research-based 
document developed by the SEA, key school improvement specialists, and select educators 
across the state in 2005, provides an organizing framework and touchstone for these systems. 
The SIF is organized using strands, standards, and benchmarks. Five key strands identify general 
areas of focus: teaching for learning, leadership, personnel and professional learning, school 
and community relations, and data and information management. Each strand contains two to 
three standards with corresponding indicators and rubrics that specify best practices and guide 
the state's accreditation and school improvement process. Under state law, all schools and 
school districts are required to prepare three-to-five year SIPs. The state school accreditation and 
accountability system is based on student achievement (proficiency rates) and schools' progress 
in meeting process indicators aligned with the SIF. 
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State B's school improvement system targets Title I schools that have not met their performance 
goals. The type and level of SSOS support changed over time as a result of state law, 
implementation of Title I's SIG program, State B's ESEA Flexibility waiver and changes in the 
state's testing system. Some of these changes, such as the ESEA waiver and testing changes, 
occurred toward the end of our study. The SSOS was designed to assist Title I schools that had 
not met their AYP targets in mathematics and/or reading, and had been identified as in need of 
improvement, corrective action or restructuring. It comprised overlapping but different support 
networks for identified schools as well as learning opportunities and guidance for leadership 
and staff in those schools: a principal fellowship at a state university; a network of school- 
based leadership coaches; a network of school-based content, instructional, or data coaches (for 
guiding and supporting principals and teachers improvement processes); a network of School 
Support Teams; and a network of school auditors (to monitor the "continuous improvement 
cycle" providing "snapshots" schools' improvement status). The SSOS also required participating 
schools through their districts to document alignment of curriculum with state content standards 
through the Survey of Enacted Curriculum. While the SSOS goals aimed to improve classroom 
instruction, the components primarily targeted school-level leadership-principals, assistant 
principals, and teacher leaders. 

As required under the state's 2009 reform legislation, the 5% lowest-achieving schools must 
adopt one of the four federal Title I SIG intervention models. These "priority" schools have the 
option to apply for SIG funding. If awarded this federal grant, the school does not receive support 
through the SSOS. Instead, it follows a state- approved plan with the option of purchasing SSOS 
supports using its SIG funds. These schools, however, must have a school support team in 
place. External administrators hired by the state association of AEAs facilitate and monitor SIG 
implementation. Other Title I schools that have not met their AYP targets have access to the 
SSOS resources, based on demonstrated need. 4 

The School Support Team in 2011-2012 consisted of an AEA school improvement facilitator, 
a district representative and, for schools that remained in the SSOS after the first year, an 
instructional leadership coach, a content coach, and an SEA field services representative. The 
School Support Team reviews and analyzes all facets of the school's operation, including the 
design and operation of the instructional program; assists the school in identifying methods 
for improving student performance; and, facilitates and monitors implementation of the SIP 
and cycle of continuous improvement. The leadership coach worked for up to 1 00 days with the 
principal and other school leaders to support implementation of the SIP and an Instructional 
Learning Cycle (ILC). The ILC process for instructional improvement provides teachers, coaches 
and/or principals with opportunities to reflect on the quality of instruction in classrooms, and 


4 The State's ESEA Flexibility Waiver (approved in mid-2012) creates three categories of schools: priority 
schools as currently defined by state law; focus schools, defined as 1 0% of schools with the largest gap between 
their high and low performing students; and other Title I schools not making AYP. 
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links instructional strategies and school-wide instructional priorities to the state's academic 
standards. Instructional coaches were hired through a district's local AEA. Under its ESEA waiver, 
State B is replacing school-based Instructional Leadership Coaches with Intervention Specialists 
(for priority schools) or District Improvement Facilitators (for focus and other identified schools). 

State B has also begun, more recently, to focus on the district's role in supporting school 
improvement. As described in more detail in subsequent chapters of this report, this shift was 
driven by several factors. First, feedback from the field, as well as the external evaluation of the 
SSOS, identified lack of district capacity and problematic district practices as inhibiting the school 
improvement process. Second, the State Board of Education set new proficiency cut scores to 
reflect the state's more challenging college and career-readiness standards. This, coupled with 
changes to the state's SSOS under its ESEA waiver, will require intervention in far more priority 
and focus schools than under the former AYP system. Third, the SEA sought out and several of 
the SEA's external partners shared research, tools, and strategies for building district capacity 
in school improvement through district intervention specialists, teams, district improvement 
facilitators, and district-level toolkits. 

Funding of the School Improvement 
System 

State B's school improvement work was funded primarily with federal Title I funds earmarked 
for school improvement generally and SIG schools in particular. SEA Title I funds supported: 
positions in the Department of School Improvement; most components of the SSOS, such as the 
leadership coaches, instructional, data and content area coaches, and school auditors; and, the 
SEA's partners in the SSOS (such as the AEAs and state universities). Title I schools in corrective 
action or restructuring could also apply directly to the SEA forTitle 1, 4% set aside SIGs ($30,000 
to $40,000 per school based on enrollment), and AEAs could do the same for grants to provide 
technical assistance to schools participating in the SSOS. SIG-funded schools used these funds 
to purchase the services of School Support Teams and other SSOS services. 5 


5 Under its ESEA waiver, focus schools will be expected to use their district and school Title I set-asides to 

support improvement activities. 
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State C 


Context 

Legal context 

The governor of State C appoints members of the State Board of Education, with the approval of 
the legislature. He also appoints the secretary of education and the commissioner of education. 
The secretary plays a policy advisory and budgetary role, while the commissioner has authority 
over the day-to-day direction of the SEAand implementation of State Board of Education policies. 
State C operates parallel state and federal accountability systems, although elements of both 
systems were merged after the 201 2 ESEA flexibility waiver. The state accreditation system holds 
schools and school districts accountable for a set of input and outcome standards that include 
minimum staffing, coursework and facilities requirements as well as student performance on 
two subjects not included in federal accountability. Although state performance standards have 
steadily increased, the federal system holds schools and school districts accountable for higher 
levels of student performance and for subgroup performance. As a result, many more schools 
and districts meet state accreditation standards than federal accountability targets. 

Political context 

State C has a long history of standards-driven education reform, which has been embedded in 
the state's accreditation system for many years. Two successive governors have appointed the 
same commissioner of education, who has been in the SEA for more than 35 years. The state 
has used its ESEA waiver to reconfigure and connect its dual accountability systems and provide 
focused support to low-performing schools. 

Fiscal context 

Like the States A and B, State C balanced its budget during the recession through a combination 
of budget cuts, increased federal aid under the American Recovery and Reinvestment Act, and 
the use of rainy day funds. Appropriations for the SEA fell 22% between 2008-09 and 2010-11, 
resulting in fewer authorized positions and, due to hiring freezes, many more unfilled positions 
as staff left the SEA. 
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Organizational Structure and Capacity 

Organizational structure 

State C's SEA is composed of seven departments: Assessment, Administration, Policy, Curriculum 
and Instruction, Special Education and Student Support, Educator Preparation, and Career 
Education. The heads of these departments report directly to the commissioner. (See Figure 
2.3.) 

Accountability policy responsibilities are split between the Office of Program Administration in 
the Department of Curriculum and Instruction (federal accountability) and the Department 
of Policy (state accountability). The Office of Program Administration also administers all ESEA 
programs. One explanation for this division of responsibility was "just because of size." 

The state's school improvement programs are housed in the Department of Assessment. The 
Office of School Improvement was located in the commissioner's office until 2007. The Office 
of School Improvement's current location with Assessment was driven in part by a desire to 
protect its focus on lowest-performing schools. As an SEA staff member explained: "School im- 
provement is an important task... and it didn't get muddled down or watered down in [another 
office addressing] the 90%....We kept our mission without having to displace it into another silo's 
mission." The School Improvement office is responsible for improvement activities for schools 
identified under both the federal and state accountability systems, including school- and dis- 
trict-level reviews, school improvement planning, and innovative programs. More recently, the 
School Improvement office took on responsibility for helping districts conduct teacher evalua- 
tions as the Department of Educator Preparation did not have the capacity to provide this kind 
of technical assistance. 

The Assessment Office manages all of the state's statewide testing programs and operates an 
Education Management Information System [EMIS] that contains information on enrollment, 
demographics, and student achievement. Districts enter student information and the SEA 
provides assessment data. Districts determine the way they organize and use the EMIS data. 
Recently the SEA made a web-based data analysis and reporting tool available to schools and 
districts to facilitate their access to and use of the EMIS. A data specialist in the assessment office 
helps to facilitate data use by working with districts and school improvement coaches on how 
to use the EMIS to look at the district data, individual student data, or data by teachers. He also 
prepares data tables for the School Improvement office when they meet with school districts. 

Unlike our other two study states, State C does not have a system of AEAs or Regional Assistance 
Teams to deliver technical assistance and support to schools. However, seven universities host 
assistance centers for special education services, and the School Improvement office sought to 
coordinate some of their supports for identified low-performing schools. 


34 


CONSORTIUM FOR POLICY RESEARCH IN EDUCATION | Cpre.org 


SEA Context, Organization, and School Improvement Strategies and Delivery Systems 

Figure 2.3: State C 
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The commissioner of education meets weekly with his executive team which is composed of the 
assistant commissioners heading the seven departments. One respondent noted that these 
meetings are an important vehicle for communicating across the departments, and through 
which the directors of offices can make requests: "Everyone's going to have input in terms of 
what's going on, and the commissioner is going to say, 'This is what we need to do.'" There are 
no regular meeting routines at the director level, but as one director stated: "So long as I can 
interface with other directors, I get my work done. And I feel welcome in other directors' offices." 
Another director mentioned: "It's not a real formalized process here. It just depends on what 
the need is." 

Between our first and second round of data collection, the Office of School Improvement created 
an intra-agency team that meets monthly to coordinate the provision of professional develop- 
ment and other resources the SEA sends to low-performing schools and school districts. This 
effort is meantto streamline communications and to keep from providing guidance at cross-pur- 
poses. Theteam is composed of directors and staff from School Improvement, Special Education 
and Student Services, Assessment, Curriculum and Instruction, Title III, and Educator Prepara- 
tion. As one SEA respondent explained: 

Once we receive a technical assistance request form from a school district, an individual 
school, or a district liaison, then the intra-agency team will review that request, and 
then we will determine what types of professional development or resources that will 
be shared with the person or persons requesting assistance.... we're only going to send 
one person into a school district, instead of several people. 

In addition, for two years a cross-agency team from State C participated in the Center for 
Innovation and Improvement's Academy of Pacesetting States, with staff from the special 
education, Title I, ELL and School Improvement offices. 

SEA capacity 

In 2010-11, State C's SEA had about 250 employees, including consultants as well as 
professionals. Respondents reported that the SEA lost substantial internal expertise in such 
areas as assessment, curriculum and instruction, special education, and school improvement. 
Two factors led to this loss: 1) reductions in staff due to budget cuts, and 2) staff turnover due 
to an inability to provide competitive salaries. For example, the SEA merged what had been 
separate offices for elementary and secondary education within the Department of Instruction, 
cutting one director's position in the process. The elementary school director was elevated to 
oversee the new office, but acknowledged that he had no experience and expertise in secondary 
education-a challenge given new state graduation standards going into effect, and more 
high schools being identified for improvement assistance and not meeting state accreditation 
standards. The SEA had very limited research capacity, which they lost entirely during the 
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course of our fieldwork when the research director left this state-funded position (see below). 
A few respondents felt, however, that the down-sizing of the SEA had contributed to greater 
collaboration across departments and offices. 

Staff turnover was another challenge for the SEA. Staff in departments such as Special Education 
left for higher paying positions as directors in local school districts. People trained in assessment 
moved to high-paying jobs in testing companies, and the School Improvement office could not 
afford full-time salaries for individuals with experience in school improvement. As a result, the 
Assessment Office hired individuals "out of the classroom" and trained them to be assessment 
specialists in the SEA. The Office of School Improvement hired part-time staff to work at the 
SEA or as school-based coaches, and external partners and relationships filled other gaps in 
expertise (see Chapter 5). One respondent noted the positive side of this arrangement: "We 
have found, over time, that [we are ] actually better able to get better staff by hiring part-time 
people, because retired principals and superintendents are often looking for part-time work and 
are happy to work for [us] as a contractor." Similarly, several respondents suggested that the 
reduced size of the SEA actually facilitated communication and improved collaboration. One 
assistant commissioner commented that leadership is more readily accessible, and more willing 
to offer support. 

Research capacity in the SEA 

The SEA has very limited internal research capacity. State C had one person, a research director, 
who was housed in the commissioner's office. She served as an internal consultant on research, 
data use, identifying indicators for success, and brokering research contacts. The director helped 
staff put out RFPs for evaluations, worked with SEA staff on interpreting data for policy initiatives, 
assembled research resources, and helped staff develop tools, such as an early warning system 
for students at risk of dropping out. She also oversaw legislatively mandated research studies. 
This director left in 2012 and, because of state budget problems, the SEA did not plan to fill the 
position. 

School Improvement Strategies and 
Delivery System 

The Office of School Improvement provides the assistance and supports for schools and districts 
identified under either the federal or state accountability system. There is also an elaborate and 
long-established infrastructure of supports for special education, and special education staffs 
have engaged with school improvement in the Pacesetter Academy initiative and in other ways. 
At the behest of State C's governor and State Board of Education, in 2002 the SEA developed a 
strategy to assist schools not meeting state or ESEA performance requirements. At that time, 
the state's technical assistance heavily focused on the school, and emphasized curriculum 
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alignment. Although many sanctioned schools were successful in meeting standards, a point of 
pride often repeated by the SEA staff in interviews, the numbers of schools that were identified 
or failed to meet federal performance expectations began to jump exponentially in 2008-09. 
Between 2002 and 2008-09, the number of identified schools hovered steadily between 30 
and 36 schools. In 2009-10, that figure jumped to 73 new and returning schools in 41 districts, 
and by 201 0-1 1 included 174 schools in about half of the districts in the state, due to rising 
AYP targets. The SEA anticipated this number might more than double again, in part due to 
the implementation of new assessments. In addition, because the state has ratcheted up the 
graduation and completion rate required to meet state accreditation, it was anticipated that 
many more high schools would fall under conditional accreditation status. 

As a result of this and anticipated growth in identified schools, in academic year 2010-11 the 
SEA began to focus more attention on making the district a primary partner in the improvement 
process. Multiple respondents identified this new effort with districts as the key change in their 
program's approach to support, and an evolving effort. 

In order to get this district approach we just didn't say overnight, 'This is it, we're go- 
ing to it.' I wanted a way to look at how to work with the district effectively. And so, we 
started working with the schools that were at the very bottom and with their districts in 
building district support for those schools, and then bridging it with school support. 
And so rather than have a school coach, they really had a district coach. 

The SSOS in place during our study included the following elements: 1) school- and district-level 
school improvement planning and academic review processes; 2) coaches in schools and school 
districts requiring assistance; 3) an electronic platform for school improvement planning; and, 
4) extensive professional development through face-to-face and electronic venues. Where 
appropriate, we have included changes incorporated in the state's ESEA waiver. 

School and district improvement planning 

Improvement planning is a key requirement for both districts and schools and is common for 
all regardless of accountability mechanism or status. Both the district and schools select from 
a set of indicators to develop improvement plans, progress towards which is subsequently 
monitored by SEA and its partners, and/or district staff. Indicators are based on the national 
Center for Innovation and Improvement's (CM) Handbook on Restructuring and Substantial 
School Improvement (Walberg, 2007). In 2005, the SEA roughly halved the original indicators 
from the handbook for districts and schools (from approximately 40 for districts to 20, and from 
160 for schools to 82) to help schools and districts better establish priorities and action steps. 
Within this smaller group, districts and schools make choices of where to focus based upon 
their needs. For schools, these encompass 1 1 areas that range from Formative Assessment and 
Differentiated Instruction to Team Structure and Principal Role. 


38 


CONSORTIUM FOR POLICY RESEARCH IN EDUCATION | Cpre.org 


SEA Context, Organization, and School Improvement Strategies and Delivery Systems 


Academic review teams, composed of SEA contracted staff and district administrators, are as- 
signed to schools that do not meet state accreditation standards; that is, they are accredited with 
warning or provisionally accredited. These teams examine the systems, processes, and practic- 
es that are being implemented at the school and district levels that form the basis of the SIP. 
They outline the indicators that schools and districts must look for in terms of alignment, use of 
time, use of data in instruction and planning, school-based professional development, SIP, and 
research-based interventions if the school did not meet its targets in ELA or math. Academic 
review team reports provide the school and the district with Essential Actions that can be used to 
develop or revise and implementthe mandated three-year SIP. 

Title I and non-Title I schools that do not meet their annual measureable objectives, but are not 
classified as priority (the 5% lowest-achieving schools) or focus schools (the 1 0% of schools with 
the largest gap between their high- and low-performing students), must develop SIPs, in con- 
junction with district staff, using the state's electronic SIP In distar® system. The SEA is ultimately 
responsible for monitoring these plans. 

Coaches 

Anticipating larger numbers of state- or federally identified low-performing schools, and 
recognizing that districts needed to be more engaged in the work, State C shifted its resources 
from providing school-level coaches to assigning coaches to district-level teams. 

With the ESEA Flexibility Waiver, the state's coaches introduced a strategy to encourage districts 
and "focus" schools (e.g. schools with a large gap between the lowest- and highest-achieving 
students) to adopt and more effectively implement, evaluate or modify targeted student 
interventions. Districts with focus schools must also assign a coach to each of these schools. 
The district team must interface monthly with the school team, and the coach, often a retired 
principal or superintendent, sits on that interface. The coach examines what strategies the 
district brings to the school, trying to identify missing pieces. Some schools and districts that 
fail to improve are assigned an auditor who more closely monitors whether schools and districts 
have implemented recommended changes. The SEA partnered with a leadership foundation 
created by one of the state's professional membership associations, to identify auditors and 
coaches and provide on-going training to school principals and district administrators. (This 
organization trained and provided coaches prior to the waiver.) A local university and other 
external partners provide training to these coaches as well. 

Under the Title I SIG program, "priority" schools were reguired to adopt one of the four federal 
intervention models, and to work with an external provider (a Lead Turnaround Partner). State 
C identified and procured a pre-approved list of four such partners, although districts may opt 
to procure a different organization. SEA-assigned facilitators monitor the implementation of 
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priority school's reform program and request support as needed. Support may come from SEA 
staff, educational partners, or a Differentiated Technical Assistance Team that may provide off-site 
(web-based) or on-site technical assistance. 

Electronic platform for SIP 

As referenced above, the SEA uses the web-based planning and support tool, Indistar®, to guide 
the development of and monitor school and district improvement plans. The system links the 
school, district, and SEA so that reports may be submitted electronically to the SEA and state 
and district coaches can continuously review and comment on the work of the district and 
school teams. Districts and schools use Indistar® to guide and upload their improvement plans 
and subsequent progress and other required reports. Indistar® incorporates many supports 
for a continuous improvement process (assessing, planning, implementing, monitoring). 
For example, it includes Wise Ways, a compendium of research that supports the different 
indicators and provides clarification about the different indicators. It provides video modules 
demonstrating the practices, and a technology to enable SEA contractors to offer feedback to 
their assigned district and school teams in real time. One component of Indistar® allows for 
virtual coaching through the Coaching Comments feature, which enables the SEA contractors to 
offer feedback to their assigned district and school teams. 

Professional development 

The SEA and its partners provide professional development through several mediums. For 
example, the School Improvement office requires schools and districts to participate in frequent 
webinars (some strands are required monthly, others, quarterly). In these webinars, school 
improvement staff worked with the State C leadership foundation coaches and faculty from a 
state university to present information, discuss improvement planning processes and strategies 
being implemented at the local level, and focus on common issues. The SEA adopted teacher- 
leader training based on materials primarily from the Cll's Mega System (Redding, 2006), with 
a focus on differentiated instruction. They contract with trainers who "have State C's spin on 
teacher-leader training." Principals and teachers attend four trainings throughout the year, and 
are then expected to train others in their building. The SEA also uses the Southern Regional 
Education Board's leadership modules in teacher-leader training, and sites also read Guskey's 
Mastery Learning. Formative assessment training is also a major focus of interventions, and 
training is provided by Editure (formerly TeachFirst) and the CM. 
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Funding of the School Improvement 
System 

State C's school improvement activities were supported primarily with federal Title I funds 
earmarked for school improvement generally and SIG schools in particular. District activities 
were supported through theirTitle I set-aside funds. 

Looking Across the Three States 

There are many similarities and some differences in the organizational structures, school 
improvement policy contexts and delivery systems, and capacity of the three SEAs in the study 
that could influence how they search, incorporate, and use research and other types of evidence 
in their school improvement strategies. 

Organizational Structure 

The formal structure of an organization delineates the roles and responsibilities of, and lines of 
communication among, its staff. As we noted in the Introduction, SEAs have been criticized for 
their segmented structures that potentially limit communication across different educational 
programs and functions. The three SEAs we studied remain organized largely by function, 
with separate departments for special education, career education, teacher policy, assessment, 
and in States A and C, for curriculum and instruction. (The Curriculum and Instruction Office 
is housed within the School Improvement Department in State B.) School improvement, 
the focus of our study, is also a stand-alone department in States A and B. (It is located in the 
Assessment Department in State C.) Interestingly, Title I, previously a "poster child" for siloed 
SEAs, is embedded within other departments: School Improvement in State A; Field Services 
and School Improvement in State B; and, Curriculum and Instruction in State C. Accountability 
is also housed in different places across the three SEAs: with Assessment in State B; in the Policy 
(state accountability) and Curriculum and Instruction (federal accountability) departments in 
State C; and, as a separate department in State A. 

The primary mechanism for cross-agency communication in all three SEAs was regular senior staff 
meetings that included department heads, deputy commissioners and the commissioner. State 
C had established an intra-agency team to coordinate the provision of assistance services to low- 
performing schools and school districts, while State A had an intra-agency team coordinating 
data collection and analysis. All three SEAs also created ad hoc across-agency teams to respond 
to specific policy issues. In Chapter 3, we examine whether, and to what extent, these formal and 
informal organizational structures constrained and/orfacilitated communication and the flow of 
information across the SEAs. 
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School Improvement Policy Contexts 
and Delivery Systems 

While all three states incorporated federal provisions in their accountability systems, and had 
developed frameworks that guided their school improvement activities, the three states differed 
in the design and focus of their school improvement assistance programs. State A had a long 
history of supporting school districts with low-performing schools, while States B and C worked 
directly with their lowest-performing schools. At the time of our study, however, these latter two 
states had begun to include districts in their school improvement strategies. They, like State A, 
recognized that strengthening district capacity was necessarytosupportand sustain school-level 
improvement efforts. We found that these differences in the developmental stage and focus of 
their school improvement strategies had implications for the kinds of research and other forms 
of evidence that the three SEAs sought and used, as well as for who they turned to outside of the 
SEA for guidance and support (see Chapters 5 and 6). 

SEA Capacity 

All three SEAs were affected by the downturns in the national economy. All three states plugged 
holes in their education budgets through a combination of increased federal aid, additional 
state taxes, and budget cuts. Budget cuts hit the SEAs in States B and C the hardest, resulting 
in reductions in staff and unfilled positions, and in the loss of expertise in areas like secondary 
education, and, in the case of State C, assessment and special education. Key leadership in the 
School Improvement and related departments was generally not affected, however. These staff 
had served in their SEAs and current positions for several years, and were viewed throughout 
their agencies as capable and knowledgeable. Budget constraints, however, led all three SEAs 
to rely more heavily on consultants to deliver technical assistance to low-performing schools 
and school districts, and on external organizations to train and support these service delivery 
personnel. 

Probably the greatest difference in capacity across the three SEAs was in the area of research. 
State A had a robust Research Department that supported offices throughout its agency. Several 
senior staff in State B's SEA had Ph.D.s, and a small research and evaluation unit was housed in 
the Accountability and Assessment Department. In contrast, State C had a one-person research 
office housed in the commissioner's office, a position that was eliminated when she left the SEA. 
As we see in subsequent chapters of this report, SEA capacity was a major factor in the extent to 
which, and how, SEA staff used external organizations to support their work. 
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CHAPTER 3 

The Flow and Organization of 
Information in SEAs Work and 
Knowledge Networks 

Lead Author: Carol Barnes 


Unlike other forms of capital, social capital inheres in the structure of relations 
between persons and among persons. It is lodged neither in the individuals nor in 
the physical implements of production (Coleman, 1 990, p. 300). 

As discussed in Chapter 1, the formal organizational structure of most SEAs has long been 
criticized for its hierarchical and segmented or "siloed" nature, and itsfocus on federal compliance 
instead of on guidance and support for schools or districts. In her research comparing two SEAs, 
Lusi (1997) argued that non-hierarchical, less-segmented management structures could help 
build internal and external connections among SEA actors and produce more effective, adaptive 
organizations. Supporting complex school reform, she argued, was a newly added and very 
different role for SEAs that would require flatter, more integrative organizational structures more 
conducive to coherent improvement policies and the flow of knowledge. Other organizational 
research and theory, though not conducted orapplied inthe samesetting, confirm and elaborate 
Lusi's arguments (Dutton & Heaphy, 2003; Feldman & Rafaeli, 2002). Organizations are able to 
bring to bear more varied, but relevant expertise and knowledge on common problems through 
internal and external professional connections (Weick & Sutcliffe, 2001; Wenger, McDermott, & 
Snyder, 2002). 

While sociologists have studied these kinds of connections, known as "social networks" for 
years to understand the diffusion of knowledge and innovation within and across organizations, 
scholars have only recently applied a social network perspective in education settings, primarily 
in schools and school districts (Daly, 2010). A few researchers have used social network theory 
and methods to study state education policy networks, most notably Miskel and his colleagues 
(Miskel & Song, 2004; Song & Miskel, 2005) who employed these methods to describe 
cohesiveness and identify the most central, influential or "prestigious" actors in state reading 
policy networks. But we know of no studies of SEAs using these lenses to explore the exchange 
of knowledge, and the expertise staff search for, acquire and use for their school improvement 
strategies. 
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Both the social network perspective and more recent conceptions of knowledge utilization argue 
that individuals or other social units are embedded in formal or informal relations that can shape 
norms, generate social capital, and promote the exchange and use of knowledge resources or 
expertise (Coleman, 1988, 1990; Frank, Zhao, & Borman, 2004; Penuel, et al., 2009; Putnam, 
1 995). From this view, SEA actors or even offices are interdependent rather than autonomous 
(Miskel & Song, 2004). Collaborative structures and centrally positioned, influential knowledge 
brokers can open channels for the flow of information and ideas across isolated departments 
and program areas, and even across organizational boundaries (Cross & Parker, 2004). But 
brokers and network connections can also create obstacles for the productive use of expertise 
if the flow of information is diluted or disrupted due to long chains of connections, overloaded 
staff, gatekeeping or turnover among key actors (Burt & Minor, 1 983; Daly, 201 0; Scott, 201 3). 

While the organizational structures described in Chapter 2 delineated the formal roles of SEA 
staff, in this chapter we also look at the more informal organizational arrangements reported 
by SEA staff in their daily work, and, importantly, as they seek, provide and acquire knowledge 
resources to inform school improvement policies. In their study of schools, for example, 
Frank, Zhao, & Borman (2004) defined social capital as actors exchanging resources, such as 
knowledge or expertise, through interactions that are not mandated by a formal structure. 
They operationalized these social resources as interactions through which such resources are 
conveyed-advice or help networks, and talk or communications networks. Similarly, Spillane et 
al. (2008) explored the designed organization of schools as represented by formal leadership 
positions, but their network questions focused "on the organization as lived and allowed for the 
emergence of both formally designated and informal leaders (p. 203)." 

We apply this perspective to this study: the work communication structures of SEA staff involved 
in state school improvement strategies; their knowledge and advice networks; network 
properties, such as size, strength of relationships, density, and degree of centralization; and, 
the identification of the most central knowledge brokers, or influential knowledge sources, in 
SEA networks. In this chapter we address in part, two of our study's research questions: 1) Flow 
are SEAs organized for work and for managing research and expertise in their improvement 
strategies for low-performing elementary schools?; and, 2) Flow do SEAs search for, provide, 
and acquire research-based knowledge (RBK) to support improvement for low-performing 
elementary schools? More specifically, how do the structure, strength or other properties of 
SEA's knowledge networks support or impede the flow of research and innovation in these 
organizations? We examine the qualities of these networks, including social capital, in Chapter 
4. 
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Data and Analysis 

We used several kinds of qualitative and quantitative evidence to investigate SEAs internal and 
external networks related to school improvement efforts. We employed UCInet (Borgatti, Everett, 
& Freeman, 2002), a network analysis software program, and descriptive statistics to analyze the 
size, density, centralization and other properties of whole networks, and to analyze patterns of 
cross-department or within-department interactions. UCInet and qualitative data allowed us to 
investigate the relative importance or centrality of individual actors or offices within the networks, 
and to explore "ego networks" at the interpersonal level (direct ties) for focal individuals (see 
Chapter 4). 

Among other questions, the survey asked with whom respondents discussed work issues (their 
work network), and with whom they interacted to acquire three types of knowledge related to 
school improvement: RBK, other evidence/data (EBK), and practitioner advice. For each of these 
four networks, the survey also asked questions about the strength of respondents' connections 
to the people they named as resources. Ou r interview protocol also asked network q uestions and 
included cognitive prompts about work communication, and interactions around useful types of 
knowledge. (See Chapter 1 for a description of the survey sample and the survey and interview 
questions.) In this chapter we focus primarily on the research and practitioner networks, using 
the work network as a point of comparison for some analyses. 

We used several methods to analyze the survey data. First, we created an N-by-N directed data, 
matrix for the four networks in each state using UCInet 6.391 (Borgatti, Everett, & Freeman, 
2002). 1 'N' is the number of SEAsurvey respondents who self-identified as working in the 
domain of school improvement, or, in the case of State A, offices, in each matrix. For socio- 
grams and some other analyses, N also includes SEA staff identified by respondents, as well 
as identified external organizations and individuals. Matrices allowed a directed connection 
between individuals to be identified through the presence of a value (which could represent 
the presence or absence of a tie or a measure of the strength of that tie). The socio-matrices in 
UCInet for all three of the knowledge networks and the work network served as the base for 
visualizing the network structures and actors' positions in them through sociograms or maps 
of each network. We used NetDraw, a network visualization software program to create these 
maps. 


1 In asymmetrical or directed data, the direction of the tie is important and the tie itself does not convey a 

symmetrical relationship; for example, one individual can seek out information from a second individual, but the 
other does not seek it from the first. Similarly, one can provide information to another while the second does not 
provide information to the first. 
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Sociograms, or maps, for each network show the actors (in States B and C) or offices (in State 
A) represented as the nodes or points on the sociograms, and the connections among them 
represented as the lines or "ties." (See Figures 3.1 through 3.15 at the end of this chapter.) The 
strength of relationships between and among the nodes is shown by the darkness and thickness 
of the lines. (See Figures 3.2, 3.7 and 3.1 2 for examples of the research network tie strength). 
The nodes are color coded to show SEA actors' department or office assignment. Light green 
nodes in all three states, for example, represent the School Improvement Department, while 
dark green shows Curriculum and Instruction Departments (or in the case of State B, the office 
within the larger School Improvement Department). Black shapes represent the types of external 
intermediaries involved in the networks. Arrows show the direction of the lines as an individual 
or office can be the originator or recipient of a tie or connection. The former is considered an 
"out-tie" from the actor, while the latter is considered an "in-tie" to the actor. The sociograms 
can show patterns in the direction of "in-ties," "out-ties" or reciprocal ties, representing the 
connections from, or to, other SEA staff and departments, and to external intermediaries. 

Centrality 

Sociograms provide an intuitive method for looking at the configuration of the information 
networks; for example, the cross-office or within-office ties that can integrate or separate SEA 
departments, or the SEA's connections to external intermediaries. Moreover, sociograms are 
intuitive because the most well-connected and influential individuals and offices are located 
in a more central position in the sociogram, while the less well-connected are located on the 
periphery of it (see for example, Daly, 2010; Song & Miskel, 2005). We used several measures 
of centrality to identify these central actors in addition to locating them on the sociograms, more 
specifically Freeman's point centrality measures-degree centrality and betweenness centrality. 
We relied on our sociograms to identify the more influential external intermediaries. 

Degree centrality is the number of other actors directly connected to a focal person or office. 
The standardized degree centrality is the proportion of an actor's direct connections to others 
out of all possible connections, if the node were directly connected to every other node in the 
network (Burt & Minor, 1 983; Miskel, et al., 2003; Scott, 2013; Song & Miskel, 2005). Our data 
are directional because we have an interest not only that a connection exists between actors, 
but in the direction of their ties in terms of those who are searching for information, those who 
are important sources of information, and those who are both searching for and providing 
information. Therefore, we measure "in-degree" as well as "out-degree" centrality (Knoke & 
Burt, 1983; Scott, 2013). The in-degree centrality of an actor represents the total number of 
individuals who turn to that actor for knowledge and advice, while out-degree centrality is the 
total number of individuals to whom the focal actor turns for knowledge and advice. 
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We conceptualized the most central, well-connected network actors with many in-ties and out- 
ties as "knowledge brokers." The most influential knowledge brokers were at least one standard 
deviation (SD) above the mean on both in-degree and out-degree centrality, while those who 
were above the mean, but not a full SD above, on both measures were considered a second 
tier of brokers. Qualitative data, along with the in- and out-degree centrality measures, show 
that their interactions with many others supported information flow, within and outside the 
boundaries of the SEAs. 

Network structure and strength 

We measured the strength of network ties, or connections, by combining the frequency of 
communication about research or other kinds of information, and the influence respondents 
perceived the resulting information to have on their work. We multiplied the frequency score by 
a percent of influence. Thus, the strength scale is a matrix ranging from a cell defined by highly 
influential and daily contact in upper left of the table (200) to a lower right cell defined by not 
influential and contacts only a few times per year (0.5). (See Appendix Table 1.4 for the strength 
matrix.) 

We also considered the size and inclusiveness of the full internal networks (excluding ties to 
organizations and individuals external to the SEA), as well as their density and centralization. For 
the research, practitioner and work networks, the internal size is the number of ties in the network 
within offices and departments and across the SEA. To examine ties between SEA members and 
external members of networks, we used one-way out-ties from our SEA respondents forming 
in-ties to external organizations, as we interviewed a subsample of, but did not survey, external 
intermediaries. 

Network inclusiveness is the number of connected points as a proportion of the number of total 
points (n-isolates/n). "Isolates" are SEA staff or offices that do not communicate with others, nor 
do other staff communicate with them. A very few cases are categorized as isolates for a particular 
network as well, if they answered a survey but did not respond to a specific network question. 
In either case, the isolates do not contribute to the connectedness or the density in the network 
and are listed in tables or shown as nodes on the left hand side of the sociograms. (See Appendix 
Table 1 .2 for survey response rates by network.) 

The density concept is one way to summarize how well-connected or cohesive a network is, 
assuming that a fully complete network would mean that every actor is directly connected to 
every other actor, a very rare occurrence. Density is related to the size and inclusiveness of each 
network and is calculated from Freeman's point "degree," meaning the total number of other 
nodes directly connected with the focal node-SEA staff or offices. Density (0-1 ) is the number 
of actual ties in the network as a proportion of the number of all possible ties, l/n(n-1 ), where 
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I equals the lines or ties in the network and n(n-1) is the number of all possible ties if every 
node was connected to every other, but itself (Scott, 201 3). Directed data density is expressed as 
ll/n(n-1 ), or the number of total pairs present divided by all theoretically possible ties. We also 
use a related measure of network cohesiveness, Freeman's degree centralization. Centralization 
shows the extent of variability in the number of connections across actors as a percentage of 
a "star network", or the most centralized, unequal network possible. In a perfect star network 
only one central figure is connected directly to every other node in the network, while others are 
connected only to the focal figure. 

The Structure and Strength of SEA's 
Knowledge and Work Networks 

Size and strength 

Multiple SEA staff in the three study states actively searched for and were receptive to research 
ideas and related information from both within and outside their agencies. About 75% of the 
staff in each agency asked their SEA colleagues for research advice, while a little less than one- 
third turned to external organizations or individuals for similar information. In each SEA, some, 
but not all, of these staff named multiple colleagues, offices and external organizations as 
sources of research information. 

In all three states we found a sizable number of internal SEA staff or office connections in the 
RBK networks, which were generally more inclusive (fewer isolated actors disconnected from 
others) and larger (more connections) than their practitioner advice networks (see Table 3.1 ). 2 
Likewise, there were more total conversations around research than around practitioner advice. 
But the practitioner network members in States B and C have, on average, stronger relationships 
than those in the RBK networks, suggesting that the members of these smaller networks have 
a greater impact on one another's work than do those in the research networks. In State A the 
strength of the practitioner and research networks are closer to the same, possibly due to a 
strong, active internal research office. Patterns in the knowledge networks thus suggest that 
while SEA members sought out and provided experiential, practitioner advice to one another, 
many also interacted around and conducted reasonably broad searches for RBK. 


2 Table 3.1 includes ties to offices, ties to external organizations, and ties to individuals that a respondent 
named as a connection although that person may not have returned a survey. 
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Table 3.1. Network Size and Strength, by Network and State 


State A 

RBK 

Practitioner 

Work 

Total respondents in each network 

95 

89 

97 

Number of isolates 

19 

27 

3 

Inclusiveness 

80% 

70% 

97% 

Number of internal ties (individuals) 

442 

291 

1,188 

Internal interactions/conversations * 

19,195 

13,570 

72,885 

Average strength of internal ties 

38.9 

41.2 

52.1 

State B 

RBK 

Practitioner 

Work 

Total respondents in each network 

121 

115 

123 

Number of isolates 

19 

40 

0 

Inclusiveness 

90% 

65% 

100% 

Number of internal ties (individuals) 

435 

205 

777 

Internal interactions/conversations* 

23,689 

13,041 

58,630 

Average strength of internal ties 

44.4 

54.4 

79.5 

State C 

RBK 

Practitioner 

Work 

Total respondents in each network 

79 

78 

80 

Number of isolates 

11 

23 

1 

Inclusiveness 

86% 

70% 

99% 

Number of internal ties (individuals) 

279 

155 

720 

Internal interactions/conversations* 

15,955 

11,820 

49,450 

Average strength of internal ties 

52.7 

71.4 

57.3 


* Total of weighted frequency scores 
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The respondents' work networks are, not surprisingly, the most inclusive networks and largest 
in terms of the number of connections (or ties) to others in each state. It follows that there is 
more conversation and interaction around work issues than around school improvement-related 
research or practitioner advice. In all three states, the connections that exist among SEA staff 
and offices are, on average, stronger in the work networks as well, suggesting the actors in these 
networks interact more frequently, and exert more influence, on average, on one another than 
do those in the knowledge networks. This work pattern is not unexpected as given limited time, 
more SEA staff are likely to be engaged in discussion about daily work issues than in the process 
of searching for and providing research to one another. 

Density 

Density scores for each state are reported in Table 3.2. Unlike Table 3.1, here the number of 
ties is limited to the survey respondents in the matrix, and the number of respondents in State 
A is reported as offices, not individuals as in States B and C. As might be expected, density 
scores within each state follow a similar pattern as reported above because the more inclusive a 
network and the more ties among actors in the network, the more dense it will be (Scott, 201 3; 
Hanneman & Riddle, 2005). In all three states, the density of the RBK networks are slightly 
higher than the practitioner networks with 1 8% of potential office connections present in State A, 
and only 1 .6% or 2.6% in States B and C respectively. 3 While these are low percentages, they do 
represent 31 0 cross office ties in the State A research network, 254 ties between staff in the State 
B research network, and 169 ties in State C, which are not insignificant considering the multiple 
isolates in States B and C. These numbers do not include connections to external organizations 
or individuals for the search and acquisition of research; these will be discussed further below, 
and in Chapter 5. The average number of research related internal ties for an office in State A 
is 7.4, while the average number of colleagues a staff member connects with about research 
is a little over 2 in both States B and C. Though the research network densities and average 
connections are only slightly higher, it is easy to visualize the denser web of connectivity in the 
larger research networks, and the sparser practitioner advice networks through our sociograms. 
(See Figures 3.6 (RBK) and 3.8 (practitioner) for State B; and Figures 3.11 (RBK) and 3.13 
(practitioner) for State C). 


3 We cannot compare density across states. The density for State A is much higher than the other two states 

because each node in the network is an office representing a collection of individuals, not simply one individual, 
and thus the network is far smaller in terms of possible nodes and ties. In practice, a high proportion of all possible 
connections in a smaller network is easier to achieve than in a large network. Given that density scores are related 
to network size and inclusiveness, the method of computing them can, but does not always, report lower densities 
for larger networks because there are more ties theoretically possible than actors can typically sustain given upper 
limits to their capacity for doing so (Scott, 2013). To visualize the difference in the size of State A's networks, see 
Figure 3.5, compared to Figures 3.10 (State B) and State Figure 3.1 5 (State C). 
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Table 3.2. Network Density, by Network and State 


State A 

RBK 

Practitioner 

Work 

Total respondents/offices in each network 

42 (offices) 

42 (offices) 

42 (offices) 

Number of isolates 

0 

4 

0 

Inclusiveness 

100% 

90% 

100% 

Density 

0.180 

0.109 

0.405 

Number of ties 

310 

188 

697 

Average degree 

7.381 

4.476 

16.595 

State B 

RBK 

Practitioner 

Work 

Total respondents in each network 

127 

127 

127 

Number of isolates 

34 

60 

10 

Inclusiveness 

73% 

53% 

92% 

Density 

0.016 

0.007 

0.032 

Number of ties 

254 

116 

508 

Average degree 

2.024 

0.913 

4.063 

State C 

RBK 

Practitioner 

Work 

Total respondents in each network 

81 

81 

81 

Number of isolates 

18 

32 

7 

Inclusiveness 

78% 

60% 

91% 

Density 

0.026 

0.014 

0.063 

Number of ties 

169 

89 

409 

Average degree 

2.086 

1.099 

5.049 
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Density percentages are sometimes underestimated in larger networks because there are many 
more relationships theoretically possible than an actor can sustain given upper limits on his 
or her capacity for doing so. 4 Still, the largest, most inclusive work networks are the densest 
networks in each state, with a larger percentage of relationships existing out of those that are 
possible than in the two knowledge networks. In the State A work network, for example, just 
over 40% of all possible ties between offices are present and the average number of ties per 
office is 1 6.6. In States B and C, respectively, 3.2% and 6.3% of potential SEA staff connections 
are present in the work networks. The average number of connections per staff member is a 
little over 4 in State B and 5 in State C. Again, it is not surprising that SEA staff invest in more 
relationships related to work issues than they do in searching for and providing research 
knowledge or practitioner advice. 

Density is expressed across all staff who reported involvement in school improvement work and 
does not distinguish among actors in terms of how deeply involved they are, or their specific 
job descriptions. Some staff may not work in positions that would require much research. 
But, given the limited time that SEA staff have at their disposal, direct connections with four to 
five colleagues is a substantial number to meaningfully maintain in the work network. In the 
research and practitioner networks, while the average degree or number of connections is about 
two, 5 we found variability in the number of relationships across actors, with a few important SEA 
staff having many connections to others while some had only one or none (the isolates). Thus, 
while there are a sizable number of direct connections in these relatively large organizations, 
they tend to be concentrated among a smaller number of key actors, a topic we discuss in more 
detail below. 

Network centralization 

The degree centralization of a network is a related lens for looking at the connectivity between 
actors or offices in the network as a whole as it compares the centrality or "connectedness" of 
the most central actor with that of all others in the network. 6 Centralization, similar to density, is 
sensitive to network size and inclusiveness. 


4 In addition, the online survey would allow a respondent to write in only three individuals for each office/ 
department he checked in States B and C, although with the possibility of checking multiple departments, the total 
number of individuals identified could still be quite high. 

5 These average density degrees include the multiple isolates who have no ties, and do not consider the 
direction or reciprocity of the tie. Below we show higher average ties between connected staff, more accurately 
including the number of in-ties and the number of out-ties. 

6 Centralization shows the extent of variability in the number of connections across actors as a percentage 
of a "star network," or the most centralized, unequal network possible. In a perfect star network only one central 
figure is connected directly to every other node in the network, while others are connected only to the focal figure. 
The higher the percentage, the more centralized the network. To calculate centralization, one subtracts the sum of 
the differences in centrality scores of each actor compared to the most central actor (C* [Z(C*- Ci)]), divides by the 
sum of differences in the most centralized star network, and turns the resulting number into a percentage. Central- 
ization = 100*((2(C*- Ci) / Max 2(C*- Ci)). For more details, see Hanneman & Riddle (2005). 
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Table 3.3: Network Centralization, by Network and State 


State A 

RBK 

Practitioner 

Work 

Total respondents/offices in each network 

42 (offices) 

42 (offices) 

42 (offices) 

Number of isolates 

0 

4 

0 

Inclusiveness 

100% 

90% 

100% 

Out-degree centralization 

71.51% 

43.78% 

60.98% 

In-degree centralization 

34.03% 

23.80% 

35.99% 

State B 

RBK 

Practitioner 

Work 

Total respondents in each network 

127 

127 

127 

Number of isolates 

34 

60 

10 

Inclusiveness 

73% 

53% 

92% 

Out-degree centralization 

10.38% 

7.27% 

10.35% 

In-degree centralization 

15.18% 

5.67% 

23.95% 

State C 

RBK 

Practitioner 

Work 

Total respondents in each network 

81 

81 

81 

Number of isolates 

18 

32 

7 

Inclusiveness 

78% 

60% 

91% 

Out-degree centralization 

11.28% 

6.20% 

18.92% 

In-degree centralization 

35.33% 

20.13% 

32.84% 


Table 3.3 shows that, across the three states generally, the SEA staff or offices in the research 
networks are somewhat more centralized around a smaller cluster of actors or a few offices than 
those in the practitioner networks, for both searching for knowledge and providing it to others.7 


7 Recall that the networks in State A are much smaller as each node on the sociogram represents an office 

(only 42), and thus would more easily achieve a high proportion of centralization than would States B and C, where 
each node on the sociogram represents a person with much higher Ns. 
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Instate C, in-degree centralization for the research network is similar to that for the work network 
(about 35% and 33 %), suggesting that influence over work and research in the SEA is somewhat 
concentrated in a smaller group of staff. (See Figure 3.11, the State C research sociogram, to 
visualize how this looks near the center of the research network.) More generally, in States B and 
Cthe search for research or ties directed out from the SEA staff (out-degree) are less centralized 
than ties directed into staff who are sources of knowledge (in-degree). This suggests more 
concentration of influence over research resources in fewer staff, while the search for research 
is more equally distributed across staff. We see a similar pattern in the work networks. The 
opposite is the case in the offices in State A where out-ties (out-degree) representing the search 
for research are considerably more centralized than the in-ties representing the sources of the 
research that is provided (in-degree). Work networks are similar. Thus, the search for research is 
more concentrated in fewer offices, while the sources of school improvement research are more 
distributed across offices. This appears on the State A sociogram (Figure 3.1 ). The many arrows 
directed into multiple departments or offices suggest that multiple hubs serve as sources of 
research, although some offices search only a few offices at a time, while a few others conduct 
broader searches. 

Centrality of staff or offices 

Staff or office centrality is a way to understand the relative connectedness and influence of each 
SEA staff or office in their networks. The most central SEA staff or offices in the networks have 
more direct connections to colleagues than others, and thus are more likely to have access to a 
wider variety of knowledge resources, as well as more influence in the network, because they are 
often important sources of knowledge and expertise. 

Means and other descriptive statistics from the distribution of the staff or office centra I ity-both 
in-degree and out-degree-provide a more detailed understanding of the search for, exchange 
and influence over research use in the networks. Table 3.4 includes only those staff connected 
within the network, not isolates. The mean number of relationships in the research network 
distribution for those staff or offices that are actually connected, even by one tie, is higher than 
the average of the two (either in-tie or out-tie) that we discussed in the density section (Table 
3.2). For example, in State B the average staff with out-ties in the research network had an out- 
degree centrality of 3.67, while the average staff with in-ties had an in-degree centrality of 4.28. 
Thus, the average staff with out-ties searched internally for information from between three and 
four colleagues. Those who were sought out for advice on research, had on average, four to five 
staff coming directly to them (not through an intermediary). State C is similar, with a 2.98 out- 
degree centrality and 4.33 in-degree centrality. These numbers do not include external search; 
that would make the average number of out-ties higher. 

For States B and C, Table 3.4 also shows low normed in-degree and out-degree centrality, or the 
percentages of direct ties present out of those theoretically possible if the average actor or office 
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were directly connected to every other node in the network. However, in practice, and in the 
context of our study, the actual number of relationships or interaction with colleagues around 
research is a reasonable number compared to the average number of ties in the practitioner 
networks, given the size of the SEAs, and in light of staff limits on time and capacity for locating 
and providing research to one another. 


Table 3.4 Centrality Distribution for SEA Staff or Offices 



Out-degree 

Centrality 

In-degree 

Centrality 

Normed Out- 
degree 
Centrality 

Normed In- 
degree 
Centrality 

State A RBK 

Mean 

10.44 

7.38 

25.47% 

18.00% 

Std Dev 

9.09 

4.84 

22.16 

11.80 

Variance 

72.23 

22.85 

429.71 

135.96 

State A P RAC 

Mean 

6.17 

5.08 

15.04% 

12.39% 

Std Dev 

5.98 

3.67 

14.59 

8.95 

Variance 

33.67 

14.25 

200.35 

84.77 

State B RBK 

Mean 

3.67 

4.28 

2.91% 

3.40% 

Std Dev 

2.81 

4.48 

2.23 

3.55 

Variance 

7.63 

13.88 

4.80 

8.74 

State B PRAC 

Mean 

1.40 

2.64 

1.11% 

2.09% 

Std Dev 

2.28 

2.01 

1.81 

1.60 

Variance 

2.83 

4.61 

3.35 

5.45 

State C RBK 

Mean 

2.98 

4.33 

3.73% 

5.42% 

Std Dev 

2.62 

5.17 

3.27 

6.47 

Variance 

7.21 

14.83 

8.16 

16.78 

State C PRAC 

Mean 

1.79 

2.78 

2.24 

3.48 

Std Dev 

2.06 

2.99 

2.57 

3.74 

Variance 

2.83 

4.61 

3.35 

5.54 
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It is important to notice that we see more variability in States B and C across SEA staff who are 
sources of information (in-degree) than among those who search for information (out-degree). In 
State B, for example, the standard deviation for search (out-degree) is 2.81 , while the in-degree 
is 4.48, and the variance around the mean for search is 7.63, with a variance of 13.88 for the 
sources of research (in-degree).There is likewise a large range in the number of connections an 
actor has when searching for research, but even higher when serving as the source for research 
to others in the networks, 1 to15 and 1 to 21, respectively. The data show similar results in 
State C, where the average out-tie and in-tie is about three and four respectively, but there is 
considerably more variability across actors for in-degree than out-degree with a variance of 7.2 
for out-degree and 1 4.8 for in-degree. There is a very wide range in the number of ties in to 
particular sources of research, from 1 to 30, and a smaller range in search, with staff turning to 
from 1 to 12 colleagues. Thus prestige, and the ability to influence research provided to others, 
is more centralized among a few key SEA staff than is the search for information in States B and 
C research networks. But consistent with our previous discussion of network centralization, in 
State A the opposite is the case where broad searches for research are more concentrated in 
fewer important offices in that SEA, while the sources for research are distributed more equally 
across offices. 

Influential Sources of Research 
or Expertise and Well-Connected 
Knowledge Brokers 

We identified the most influential or prestigious internal network members using normed in- 
degree centrality scores, and a rank order of in-ties or in-degrees. Influential participants are 
those who were highly sought after for research information; that is, they were mentioned as 
a source of advice by many SEA staff. We conceptualized the most central "well-connected" 
internal network actors, those with many in-ties and out-ties, as knowledge brokers. 8 Well- 
connected actors or offices are those who both sought research ideas and information from 
a range of sources, and, at the same time, provided information to multiple colleagues, thus 
serving as information brokers in the networks. Not surprisingly some of the most influential 
individuals or offices, those with the most in-ties, were also well connected, with many out-ties 
as well because the two measures can be highly correlated (see for example, Song & Miskel, 
2005; Burt & Minor, 1983). 


8 For our final list of key knowledge brokers we included those who were at least one SD above the mean 

on in-degree centrality, out-degree centrality, and betweenness centrality. Freeman's betweenness centrality iden- 
tifies potential knowledge brokers who may not be otherwise influential, but, more often than most, connect other 
actors or offices indirectly through the shortest path (geodesic) distance. Thus, these actors can serve as intermedi- 
aries linking SEA staff who may not otherwise be connected. We also identified and included a few actors who were 
above the mean, but less than one SD above the mean, on at least two of the centrality measures as a second tier of 
information brokers. 
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These key SEA staff or offices were in the "thick of the network relationships" in terms of the flow 
of information. On the sociograms that map each participant's location in the networks (e.g., 
central or more peripheral), the more influential and well-connected individuals and offices 
tend to be clustered more centrally, or, in the case of State A, in multiple hubs of interaction. 
For example, they appear on the research network sociogram in State B (Figure 3.7) as a more 
central, denser cluster of staff, and in State C (Figure 3.1 2) as only a very few staff, primarily one 
or two, while State A (Figure 3.2) has a few offices more centrally located than others. Notice the 
strength (darker, thicker lines) of the many ties directed into and out from these influential and/ 
or well-connected staff. 

Note also that both the research and the practitioner network sociograms (see Figures 3.3 and 
3.4, Figures 3.8 and 3.9, and Figures 3.1 3 and 3.1 4 for the practitioner networks in States A, B and 
C, respectively) show that many of the SEA staff in these networks did search externally, but most 
external sources of research and practitioner advice are on the periphery of the networks (black 
shapes primarily around the outside of the sociograms). 9 Most externals had ties from only one 
staff who was frequently not well connected into the internal network. This suggests that many 
external intermediaries were not well integrated into the flow of information internally. But in 
States B and C, a few external intermediaries were directly connected to the most central and 
influential brokers in the school improvement department, and thus appear on the sociograms 
closer than most or within the more central denser clusters of the network. 10 

In the research networks, the directors of school improvement (light green nodes) were the 
most influential and well-connected SEA staff in States B and C. In State B, for example, the 
director's standardized in-degree (in-tie) and out-degree (out-tie) centrality were both about 4 
SDs above the mean, while his betweenness centrality was 4.9 SDs above the mean. The latter 
suggests that he, more frequently than others in the network, served as an indirect connection 
between pairs of staff or parts of the network that were otherwise not connected directly. The 
deputy director's standardized in-degree in State B was 2.7 SDs above the mean. In State C, the 
director's in-degree centrality was almost 6 SDs above the mean with out-ties to others about 3.5 
SDs above the mean. This director's betweeness centrality was 4.5 SDs above the mean. These 
directors had a greater ability to influence the flow of research or other information than others 
in the network. They and some of their key staff were connected with several other influential 
and/or well-connected directors and staff in salient offices related to school improvement. 


9 We find this pattern in part because we did not survey external sources, but also because many of these 
intermediaries have only one connection to SEA staff and are not directly connected to the most influential brokers, 
which would make the external source more influential. 

10 We do not discuss the data networks in this chapter. The sociograms for the data networks in the three 
SEAs can be found in Appendix Figures 3.1 through 3.6. 
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State B 

In State B, these other influential staff included the director and other leadership from the 
Assessment and Accountability Department, shown on the sociograms as grey nodes (among 
them, the supervisor of the department and one staff in the research and evaluation office, 
shown in red), the director of the commissioner's School Reform Office, a deputy commissioner 
and his senior policy advisor (yellow nodes), the director of the Federal Program Monitoring 
Department (aqua node), and a consultant from special education (pink node). (See Figures 
3.6 and 3.7.) We also found strong connections between the Curriculum and Instruction 
Office (dark green nodes) and the School Support Services Office, both of which were within 
the School Improvement Department. Importantly, 12 of the 19 most influential staff in the 
State B research network were directors or middle managers. 11 All three of the key research 
brokers were directors. 12 Their relatively high betweenness centrality suggests that indirect 
network connections between many pairs of staff were dependent upon these brokers. More 
generally, all of these centrality scores, along with a qualitative examination of the influential 
brokers, suggest that much of the influence and many of the cross-department connections are 
concentrated in formal authority and coordinating roles. 

Our data also indicate there may be some diverse and important resources for research that 
are overlooked in the research network in this state. Eight staff in State B were at least one 
SD above the mean for search, but no one turned to three of them, all lower level staff, for 
research information. In addition, eight staff, all but one lower-level consultants or analysts, 
were conducting reasonably broad searches for research (above the mean) and no one turned 
to them for information. 13 

Finally, three external organizations-a statewide professional membership association (PMA) 
(black triangle), the state's regional CAC (black square) and a state university (black diamond)- 
were more centrally located than most and had stronger connections to the key school 
improvement staff than other external organizations in State B's research network. 14 In addition, 
theCenteron Innovation and Improvement(CII), a federal comprehensive center (a black square 
in the right, upper quadrant of the sociogram), and a national professional association (black 
triangle in the right upper quadrant) were also quite influential sources of research or expertise. 

1 1 Ten were at least one SD above the mean on standardized in-tie scores while another nine were above 
the mean in rank order. 

12 They were at least one SD above the mean on standardized in-tie and out-tie centrality scores, as well as 
normed betweenness scores, which means that relative to others in the network, they frequently connected staff 
indirectly who would otherwise not be connected directly. 

13 This finding may also be due in part to limits of the survey, which allowed respondents to write in only 
three individuals for each office or department. 

14 The external organizations are coded using the following shapes: government (square); PMAs (up- 
ward-pointing triangle); institutions of higher education (diamond); research/provider organizations (down- 
ward-pointing triangle) and other organizations/education journals (hour-glass). 
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State C 

In this SEA, the Offices of School Improvement (light green node)and Curriculum and Instruction 
(dark green nodes)are housed in different departments, and leadership from both departments, 
along with a very few staff, were more centrally located in its research network. (See Figures 3.1 1 
and 3.1 2.) It is easy to identify the most influential and well-connected information broker (light 
green node near the center of a denser cluster of nodes) in the network. There are many arrows 
directed to him, showing he is a prominent source of research, and arrows directed out from 
him to several different department colleagues, showing that he also conducts broad searches. 
While the assessment office staff were not as prominent as they were in State B (see only one 
grey node within the cluster in left half of the sociogram), the commissioner, the commissioner's 
policy analyst (yellow nodes), and the director of a very small research office housed in the 
superintendent's office (red node) were all directly connected to the school improvement 
director. The research director was also an influential source of knowledge (arrows directed into 
the red node just above the prominentgreen node). 

In addition, the director of the ESEA program monitoring department (agua node below and 
slightly to the right of the school improvement director), and, especially, the director of special 
education (pink node near the top of the cluster with multiple arrows directed toward it) are 
highly influential in the network. While the special education and school improvement directors 
are connected indirectly through an internal intermediary staff, the school improvement director 
is directly connected to a couple of special education staff. We can visualize how the school 
improvement director's higher betweenness centrality appears in this sociogram as he connects 
the pink special education nodes (upper right guadrant) and the agua nodes representing 
federal program monitoring (lower part of the graph) along with several other departments. 
Similar to State B, influence and connectedness are concentrated in formally designated higher- 
level roles. Eight out of eleven of the most influential staff were directors or middle managers. 15 
All five of the most influential staff were directors or middle managers, as were the two key 
knowledge brokers from school improvement and special education. 

Also, like State B, the search for research was more distributed across staff than were the sources 
of research, which were more concentrated in a few key influential directors or middle managers. 
But, three of the eight staff who were at least one SD above the mean for search (out-degree), 
and thus conducting broad searches for research, were not, in turn, consulted about research 
information. All three were lower level staff. No one in the network sought out research or 
information from almost half of the 1 7 staff (all in lower-level positions in multiple departments) 
who engaged in searches for research and had out-degree measures that were above the mean, 
but not by a standard deviation. 


15 Four were at least one SD above the mean, and seven more were above the mean, but less than one SD 
above the mean. 
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The sociograms again show that only a few external organizations were directly connected to the 
most influential broker and were more prominent in the research network than other external 
organizations-the Cll and the National High School Center (black squares within the denser 
part of the network), State C's regional CAC (black square), and a statewide PMA (black triangle). 
Of these, the Cll, regional CAC and professional association had very strong ties from the school 
improvement director. 

State A 

Recall that State A is a different case than the other two states as the unit of analysis is the office 
level. 16 The Research Department in State A (red node) was the most influential office, with more 
in-ties than any other (Figures 3.1 and 3.2). It resided near the center of State A's research network 
map having multiple connections, both in-ties and out-ties, to many other offices and crossing 
department boundaries in its SEA. This relatively large department plays multiple roles in State 
A's SEA: conducting research reviews for the agency; helping program offices design, procure 
and manage program evaluations; preparing data reports and briefings for accountability review 
teams; and developing analytical tools. The Accountability Department (dark grey node just 
above the red node), a well-connected knowledge broker, conducts formal reviews and monitors 
the performance of low-performing school districts. Staff in the Accountability Department 
indirectly linked staff in many other offices/departments who were not connected directly, by 
both searching for and providing research. Similar to State C, curriculum and instruction and 
school improvement are housed in separate departments, but both departments and the 
constituent Offices of the School Improvement Department (light green nodes) were influential 
hubs of activity with many in-ties. Notice in Figure 3.2 that curriculum and instruction (dark 
green nodes) forms a triad with the Research Department and the department housing the 
Assessment Office (dark grey node), but, of these, the latter had many more ties directed out 
in searches than ties directed into it as a source of research. Offices within the Curriculum and 
Instruction Department that work with district assistance teams, especially mathematics (dark 
green node in the upper right quadrant on the periphery of Figure 3.2) and literacy (dark green 
node in the middle of bottom right quadrant), have strong reciprocal ties with the regional 
assistance office (lighter green node to the left of the mathematics office) within the School 
Improvement Department. Finally, external organizations remained at the periphery of State A's 
research network, a topic that will be explained in more detail in Chapter 5. 

Overlapping networks and boundary spanners 

In addition to the sociograms and survey findings, a qualitative analysis of the ego networks, or 


16 Here and in previous analyses we present our findings for State A cautiously, as our survey sampling 
frame varied some across offices (see Chapter 1). We corroborated our survey findings with in-depth interviews and 
present here the results of triangulation across these data. 
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direct ties of the influential and well-connected individuals that we have discussed in States B 
and C, show their networks spanned unit boundaries internally and SEA boundaries externally. 
Further, nearly all interview respondents perceived these influential actors to be "idea people," 
"visionary," or institutional historians based on their years of experience in the agency or around 
particular issue areas. These were the people who "made things happen". 

They also tended to be in different, but still a few overlapping, networks, thus in part "distributing" 
the search for expertise or different kinds of information, but also communicating freguently 
with one another. For example, one respondent in State B said: 

I think what helps us is the regular communication that we have and through email, 
through meetings, these same people come to the high priority [regional field] 
meetings, they also go to school improvement facilitators network. . . So we get 
professional development there, we can network there and talk there. So I think that 
those conversations and sharing of research. . . has really helped. 

Astudy of this influential broker's direct ties showed that she and others named individuals who 
were influential in their work, and who were in the same practitioner or research networks. But 
this influential actor, and others like her, were also part of an internal cross-office group known 
as the "brain trust." She would take information from her network connections into this cross- 
office group that understood different perspectives on the problems they encountered, and kept 
each other informed. Similarly, a respondent in State C reported: " We don't know everything, 
but there are some big initiatives that we're all at the table on." Another State C respondent 
reported: "We met once in April, and then we had separate teams that would meet individually 
and then we would come back together and share our information." In State B, an influential 
school improvement actor reported: "I go to the school improvement grant meetings, they go 
to the high priority meetings so that we know what everybody's doing, that we're on the same 
page. So that's where the mentor piece came in". 

Thus, many of the most influential brokers could search internally and externally for research, 
practitioner advice, and other evidence as well as interact in networks with different foci, 
but come together to keep each other informed on some part of a state problem or strategy. 
Individuals could cross boundaries of work groups and units, taking part in many of the same 
group meetings, but with varying roles. A respondent from State C said: "The leadership for 
each of those groups is different, but the core group that works on this is pretty much the same 
people." Many influential staff knew one another, interacted in different settings and could 
coordinate their work, or keep one another informed through cross-office meetings. We discuss 
these core groups in greater detail in Chapter 4. 
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Cross-Office and Within Office 
Connections 

Contrary to the usual image of SEAs as siloed organizations, we found a considerable number 
of cross-office work and research-related network connections that could support the flow 
of information and tap a wide pool of expertise across key departments involved in school 
improvement work. While we found some isolated SEA members who did not communicate 
with anyone, we also found more lateral, flatter organizing networks for work and knowledge 
exchange, and more boundary spanning connections to external intermediaries than we 
expected given the literature on SEA structures. Indeed, respondents reported that these lateral 
connections were relatively new in their organizations and attributable to multiple factors: 
reduced SEA staffing and smaller, and in some instances flatter, organizations; state and federal 
accountability demands; competition for federal grants requiring integrated proposals using 
external expertise; and SEA leaders committed to more collaborative organizational cultures. 

For example, multiple respondents in interviews across State B and State C discussed reduced 
staffing and the relatively small size of their respective organizations as one factor that fostered 
cross office collaborations and connections to external intermediaries. A respondent with the 
advantage of institutional history in State B explained that smaller size and new leadership had 
recently created a less hierarchical and more collaborative culture in the agency: 

So it's a much smaller department. . . . Under (new commissioner) it's a flatter 
organization so you have greater access to the decision makers. . . I think because we're 
a smaller department or maybe the culture [the commissioner ] has put in place. . . 
[there are] greater opportunities to collaborate and work together. . . I think that was a 
very significant change here. 

A respondent in State C also illustrated this point about a more cohesive small staff as she 
discussed cross-office teams: 

I'm a member of all those task forces. We've been doing governor's projects for charter 
schools, virtual schools, lab schools, all at the same time. So that same core group is 
involved in all of this because there just aren't that many of us, you know? ... The 
leadership for each of those groups is different, but the core group that works on this is 
pretty much the same people because there's nobody else to do it. 

State and federal accountability demands and federal grant competitions were likewise 
important catalysts for new cross-office or external connections involving work and interaction 
focused on research. In State C, new connections among the Special Education Department, the 
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Office of School Improvement, and the regional CAC crossed internal departmental and the SEA's 
external boundaries, focused in part on developing the school turnaround or transformation 
models required by the Title I SIG program. Other federal initiatives, such as the Race to the Top 
grants, orthe American Recoveryand ReinvestmentAct state fiscal stabilization funds (SFSF) also 
stimulated cross-office search, exchange of research-based ideas, and joint work in States B and 
C. For example, in State C, a respondent said: "I know that part of the push [to develop a data 
tool for schools] was to meet the requirement to receive the SFSF funds." This initiative, which 
brought together staff from the Office of School Improvement and the Assessment Department, 
had continued as a project at the time of our data collection. In State B, multiple respondents 
discussed their process for creating an integrated Race to the Top grant proposal. One noted, 
for example: 

When we decided to go for Race to the Top, the demands of what they wanted in that 
application required that the department work collectively to get that work done. So 
teams were created for each of the core reform areas. . . .That was the first time I've ever 
seen the whole department kind of come together to craft a strategy. 

As part of its Race to the Top program, State A initiated a performance management system 
focused on the SEA's five strategic goals. The goal leader is a senior manager whose functional 
responsibility focuses on a particular goal, but other members of the team are drawn from 
programs across the agency that contribute to meeting that goal's objectives. 

Leadership in all three SEAs facilitated cross-agency collaboration by creating cross-office 
teams to share information or work on other common tasks or problems. State A, for example, 
established cross-departmental taskforces to design its system oftiered-intervention.As a high- 
level SEA staff person explained: 

One of our school standards is on tiered instruction. Through [the School Improvement 
and Accountability Departments] we convened an agency-wide group task force . . . 

across something like seven, eight units So there was a math person, somebody 

from science, literacy, special ed, English language learners, two people from social 
emotional help who had been working on positive behavior intervention supports. . 
. . Anyway, those create[d] a tiered instruction guide for the state. We never had that 
guidance for Response to Intervention. And we never had brought together academic 
tiered instruction with social emotional tiered instruction. So we wanted to bring 
those together to make a very strong statement to our districts that this is a regular ed 
responsibility. 

States B and C were active participants in an Academy of Pacesetting States hosted by the Cll 
which convened cross-office state teams on a regular basis to share and discuss research or other 
topics related to statewide systems of school improvement. 
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The "Strength of Weak Ties" 17 

While broad, more informal cross-department and external connections could facilitate the flow 
of research information and new ideas, the within office/department connections were stronger 
(AppendixTable 3.1 ). For example, in the State A research network, 114 within-department ties 
have an average strength of 71. 04, while the 305 cross-department ties have an average strength 
of 28.07.18 The pattern is similar in States B and C. The average strength of the 168 within- 
department ties in the State B research network is 75.58, while the 234 cross-department ties 
average only 23.58. In State C, 108 within-department ties have an average strength of 92.3, 
while 1 56 cross-department ties have an average strength of 27.86. The work and practitioner 
network ties follow a similar pattern across the three states, with 314 within-department ties 
having an average strength of 1 14.87 in the State B work network, for example, while 456 cross- 
department ties have a an average strength of only 26.67. This suggests that colleagues within 
the same department or office engaged in more frequent discussion or interactions, and these 
discussions had more influence on individuals' work than did the cross-department connections. 

Nevertheless, as the preceding quotation from State A showed, qualitative data suggest that 
these weaker cross-department or office ties were advantageous, not only to facilitate the search 
for and flow of research and other knowledge, but also to address duplicative inefficiencies, 
keep staff informed, and support coordination. Given time and capacity limits on SEA staff for 
maintaining relationships, weaker ties may be better suited for some of these activities as they 
do not require the kind of deep common understanding or frequent, complex and influential 
collaboration that some interdependent work does (for example, incorporating research or 
other knowledge in the process of developing, implementing and revising school improvement 
strategies as we discuss in Chapter 4). 

We also found that weak ties could represent a range of activity, from one-way cross-department 
consultation with little influence on the staff or department providing information to more 
substantial integrated work. Weak ties at the department level could also lead to more integrative 
work at the level of districts or schools. For example, we found varying numbers of relatively 
weak ties in the three states across the special education and school improvement departments 
in both the work and research networks. In State B, these ties represent relatively infrequent 
interactions and limited mutual impact on the work of these two departments. Rather, the 
connection appears to be a one-way consultation, or request for research. In State A, the weak 


1 7 We take this subtitle from Granovetter (1 973). Granover made the case that weak ties (operationalized as 
less frequent interactions) are more likely to provide information different from what we typically receive because 
these contacts move in different circles than we do. 

18 Administrative offices were not included in these analyses. 
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connections at the department level do not reflect activities that network the two departments 
in one strand of that SEA's school improvement work in districts. In State C, our qualitative 
analyses show that these weaker connections reflect coordination mechanisms and substantial 
work integrating the two departments-without intense, frequent interactions. 

In State B, for example, an analysis of the direct ties of the most influential staff shows that the 
well-connected school improvement director has one weak direct out-tie to special education 
in the work network, and three weak to moderate ties (in and out) in the research network. 
The research sociograms (Figures 3.6 and 3.7) show three special education staff (pink nodes) 
near the denser cluster of staff, all with weak ties from school improvement. One special 
education staff (pink node in upper left portion of the more central, dense cluster of staff) has 
multiple in-ties from the school improvement department suggesting, along with interview 
data, that department staff consult with this policy analyst regarding research. The work network 
sociogram (Figure 3.10) shows that most staff in the School Improvement Department (light 
green and dark green nodes) are located in the upper portion of the map and within the densest 
cluster of offices. Special education is more dispersed in the bottom portion of the map with one 
influential staff, the director, more centrally located between school improvement and federal 
program monitoring. 

This influential respondent in the Special Education Department characterizes these relatively 
weak ties between the two departments this way: 

I think we have representation on [school improvementdepartmentteams] so that we're 
aware of it [the school improvement strategy]. We hear it, but it doesn't necessarily 
translate into having an impact in terms of how we do our work. 

Fie continued to describe a one-way, but not reciprocal, impact from the tie between the two 
departments: 

For example, our representative might say, 'You need to make sure that you're coaching 
your general teachers on response to intervention,' so that they can support students in 
the general ed environment withoutthinking they need to go into this special ed world. 

And he concluded: "So it will impact their work; it doesn't necessarily come back and impact 
mine." Thus the connections appear to be primarily one-way consultations or searches for 
research from school improvement to special education. 

Though respondents in State A perceived the Special Education Department to have a history 
of being somewhat isolated from the other departments, the SEA had more recently arranged 
for some structural connections across special education, curriculum and instruction, and 
school improvement. The research sociogram (Figure 3.2) show multiple weak ties from school 


65 



State Education Agencies' Acquisition and Use of Research Knowledge in School Improvement Strategies 


improvement offices to special education, some of them reciprocated, suggesting that some 
staff in these two departments exchange research information. The work network (Figure 3.5) is 
similar, showing several weak ties between the two departments. 

But these work ties do not capture the extent of the more integrated work at the level of districts. 
State A had identified low performance in mathematics by special education students as an area 
of need in many of its districts. In response, the Division of School Improvement and Assistance 
(which houses the Special Education Department) allocated special education funds to the 
Department of Curriculum and Instruction (housed in a separate division) to hire mathematics 
consultants with special education expertise to serve on the school improvement Regional 
Assistance Teams and to support professional development. A high-level SEA respondent 
explained: 

We are using special ed funds to fund a math specialist in each [support region]They're 
tied together in a network that's supported by both SPED folks and math folks so that 
their focus is on how do you close the achievement gap? How do you make math 
accessible to kids with disabilities? . . .That means you get to work in regular classrooms 
to figure out how do you make the curriculum accessible. . . Which is providing math 
courses to everybody on the special ed nickel. And bringing the achievement gap issue 
more to the forefront of district thinking at the same time. So that's an example of how 
we're bringing [special education] into more of the mainstream of the department. 

Thus, while the direct ties between the School Improvement and Special Education Offices 
are generally weak in State A, the SEA has invested in creating stronger connections between 
these two at one important point in the system-the level of direct assistance to low-performing 
schools and districts. (These ties show up as the strong ties between curriculum and instruction 
and school improvement noted earlier). Likewise offices in the Special Education and School 
Improvement Departments appear to reciprocate some searches for research. 

In State C, based on analysis of direct ties of influential staff and the sociograms (Figures 3.1 1 
and 3.1 2), the special education and school improvementdirectors are connected only indirectly 
in the research network, through a weak intermediary tie from another special education staff 
(as noted earlier). But the school improvement director, the most influential broker, is also 
connected through direct in-ties to several (8) special education staff. This suggests that, unlike 
State B, special education staff seek out research from the School Improvement Office. But 
the director and other school improvement staff also turn to special education for research. In 
the work network (Figure 3.15), the school improvement director has five in-ties from special 
education; all but two are weak. While the work network sociogram appears dense and difficult 
to read, special education and school improvement are generally in different halves of the map, 
with the director, an influential school improvement staff, and a few others linking the two sides. 
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While these are primarily weak ties, the School Improvement Office had recently developed 
a coordinating team across the school improvement, special education instructional services 
and other offices, as well as substantial integrative instructional work aimed at the classroom 
level. For example, an influential school improvement staff person reported on a relatively new 
interagency team designed to reduce duplication and coordinate across the School Improvement 
Office and Special Education Department: 

Last year (2011-12) was the very first year that that team was in existence, and it was 
brought about from our office. ... The intra-agency team will review that reguest 
[for improvement services from districts] and then we will determine what types of 
professional development or resources that will be shared with the person or persons 
reguesting assistance. 

Actually this year we're going to combine the services that we are providing from our 
office to schools that have also been identified from the Special Education Department 
as needing assistance. . . we didn't want our services and their services to overlap, and 
so we are combining the services. 

The special education staff who connected with school improvement at the instructional level 
saw her role as improving instruction for all students, not simply special education students: 

I'm not as directly involved in that piece [coordination team]. My piece is more how 
can I help bring some support to schools in relationship to-not just special education, 
because I believe special education practices work well for all students.. .at the 
instructional level. 

She said, "I think we, at the department, pull in special education into most of the instructional 
initiatives, as well as the school improvement initiative." 

This special education instructional specialist who was linked to the school improvement director 
and some of her staff (albeit weakly) also discussed collaborative work between the two offices 
that was substantial, though perhaps not freguent with intense interpersonal interaction: 

One of the things that we did last year is a special ed institute as part of the school 
improvement institute, where we actually provided staff development, professional 
development to teachers in hopes of building teacher leaders and strategic instruction 
and differentiated instruction in the classroom, based on the data. You know, there's a 
lot of data, statewide data that shows some areas in which all students are struggling. 
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She continued: 

I think that we do stay in touch with school improvement. We're involved in the academic 
reviews that they do in schools. So we are providing that input at that level. I think the 
fact that we do planning for major activities, such as their institutes for teachers, for 
district- 1 eve I personnel, school personnel [shows we are involved]. 

Thus, in this state the weak ties between school improvement and special education have led to 
improved coordination and some substantive collaborative work. 

Summary and Conclusions 

Social network research in education settings other than SEAs suggests that informal webs of 
relationships can determine the extent to which reforms or knowledge spread and are used 
productively (Daly, 2010). Network studies in districts and schools have shown that district 
offices and site leaders tend to favor dosed, reciprocated relationships within their own group- 
district or site (Stein & Coburn, 2007; Finnigan & Daly, 2010). Using multiple kinds of data 
and analyses, both quantitative and qualitative, we found a more varied mix of weak to strong, 
within-group and cross-group, professional connections in our three SEA organizations. 

Staff in the three SEA organizations tended to be open to research ideas and other information 
from outside their boundaries and internally, with a sizable number of connections supporting 
the flow of research and other work-related information across key department or offices 
involved in school improvement work. This finding is unexpected given the literature on SEAs 
organizations that frequently depicts SEAs as more siloed or fragmented organizations. Not 
surprisingly, cross-department or office ties were relatively new in all three SEAs and were due 
to multiple factors, such as state and federal accountability, federal grants requiring integrative 
proposals, leadership, staffing reductions and more. While both within-department and cross- 
department relationships existed, the within-department or office connections were stronger 
than cross-department ties. The practitioner network connections-colleagues within the SEA 
interacting around practitioner advice-were stronger than research network connections in 
States B and C. In State A, the strength of connections in the research network and practitioner 
networks were about the same, perhaps due to the strong internal research office in that SEA. 
Across the three states, the research networks were larger and more inclusive than the practitioner 
advice networks, and staff engaged in more conversations about research than practitioner 
advice. The research networks were slightly more cohesive and centralized than the practitioner 
networks as well. But the work networks were, not surprisingly, the largest, most inclusive and 
cohesive of the networks, and they included more conversation around work issues than the 
knowledge networks. 
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While we found a sizable number of direct connections in the relatively large networks of 
individuals in States B and C, they tended to be concentrated among a smaller number of key 
actors. Further, in these states the search for research was less centralized than ties directed 
into staff who were the sources of knowledge. Thus we found more concentration of prestige 
and influence over research resources in fewer staff, while the search for research was more 
equally distributed across staff. In State A, a case where the unit of analysis was the office 
level, broad searches for research were more concentrated in fewer offices, while the sources 
of school improvement research are more distributed across offices. We also found that more 
formal hierarchy stratifies many of the internal ties. In States B and C, influence over research 
and knowledge brokering tended to be concentrated in formally designated higher-level or 
coordinating roles. A majority of the most influential staff and brokers were directors or middle 
managers. Thus while we did find more lateral, informal organizational structures in the SEA's 
knowledge and work networks, in some respects formal structures tended to determine the 
membership of these informal networks to some extent. 

This concentration of influence over research and other evidence can be both a strength and 
weakness for the flow of research and expertise in these SEAs. From one view, the ability to 
access and influence what research is provided has the potential to create obstacles because 
the few staff who are brokers may be overloaded in their work thus constraining the timely 
distribution of information. SEAs are more vulnerable to turnover and loss of a few critical 
staff who coordinate, broker, and hold a great deal of valuable knowledge resources. But from 
another view, a few brokers who have access to multiple kinds of expertise and knowledge are 
more likely to coordinate, synthesize, and otherwise productively integrate a range of knowledge 
into key decisions and policies. 

In addition, SEAs may be missing some important knowledge resources from little known 
sources in their respective networks. In both State B and State C, for example, while the search 
for research was more distributed across staff than the sources of research, no one in the network 
sought out research or information from many of the lower level staff who engaged in broad 
searches. Moreover, while many of the lower level staff in these networks did search externally, 
most external sources of research and practitioner advice were on the periphery of the networks 
for several reasons, one of which was that many were sought after from only one SEA staff who 
was freguently not directly connected to the internal influential brokers. Information in social 
networks is more likely to be distorted from its intended form as it passes through multiple 
colleagues or contacts rather than through a direct link between these lower level staff and the 
higher level brokers. Thus, these SEAs may be overlooking another set of knowledge resources, 
and some salient external intermediaries may remain disconnected from the heart of the internal 
SEA conversation about research. (See more details on external connections in Chapter 5). 
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While staff in the larger State B and C research networks, as a whole, are not tightly connected, 
sociograms, qualitative data and direct ties of influential knowledge brokers show a few 
more cohesive clusters of relationships existed. This is common in larger networks. Some of 
the influential brokers and their staff had strong connections, knew one another, had some 
common contacts and interacted in multiple different settings. The importance and productive 
nature of smaller, cohesive relationships and strong ties among network members (increased 
social capital, such as shared language, conceptual understanding, mutual trust and so on) are 
frequently written about and are something we take up in Chapter 4. 

However, we also found evidence that the brokers' many looser, or weaker, direct ties that were 
less redundant, less frequent, and less intense in exerting mutual influence over work fostered 
the flow of more diverse ideas and research within the SEAs, and, in particular, introduced these 
ideas into the interpersonal networks of these influential brokers. Respondents discussed the 
brokers as boundary spanners whose networks could cross internal units and SEA boundaries, 
yet bring groups together to share and make sense of research or other information. Given the 
constraints of time and capacity that SEA staff faced, these weak ties were better suited for not 
only the flow of ideas, but for keeping staff informed, integrating some less intensive work 
activities, and fostering some kinds of coordination. Generally these findings align with social 
network theory which suggests that weak ties between staff whose work is less similar, and who 
are more likely to work in different networks or seek research from different sources, support 
a more varied flow of information, knowledge, and expertise. Weak ties allow any given actor 
to have broader connections to many more people as the time and effort invested are limited, 
while stronger, more frequent interaction requires a greater time commitment, but allows for 
more complex, collaborative and interdependent work. 

Thus, the direct ties of the most influential and well-connected staff we identified showed that 
while broad but relatively weaker, cross-department and external research networks facilitated 
the search process and the flow of diverse ideas in the SEAs, a set of influential knowledge 
brokers who were conducting broad searches brought research and other kinds of information 
from these different sources into stronger "working groups" that we conceptualize as "core 
networks." We take up this topic in the next chapter. 
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Figure 3.2: State A, Research Network, strength of ties 
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Figure 3.3: State A, Practitioner Network, ties only 
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Figure 3.4: State A, Practitioner Network, strength of ties 
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Figure 3.5: State A, Work Network, ties only 
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Figure 3.6: State B, Research Network, ties only 
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Figure 3.7: State B, Research Network, strength of ties 
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Figure 3.8: State B, Practitioner Network, ties only 
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Figure 3.9: State B, Practitioner Network, strength of ties 



Commissioner 
| School Improvement 
| Instruction 


H Assessment/ Accountability 
| Research 
| Special Education 


| Career Education 
| Educator Prep 
Administration 


Program Monitoring 
| Early Childhood 
External 


79 





Figure 3.10: State B, Work Network, ties only 
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Figure 3.11: State C, Research Network, ties only 
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Figure 3.12: State C, Research Network, strength of ties 
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Figure 3.13: State C, Practitioner Network, ties only 
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Figure 3.14: State C, Practitioner Network, strength of ties 
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Figure 3.15: State C, Work Network, ties only 
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CHAPTER 4 

The Role of Social Networks in 
SEAs Knowledge Search and 
Incorporation Processes 

Lead Author: Carol Barnes 


Research and theory on social networks, communities of practice, and knowledge utilization 
all argue that individuals are embedded in relational systems that can shape norms, generate 
social capital, such as trust, and promote the exchange and use of resources or new knowledge 
(Coleman, 1988, 1990; Daly, 2010; Penuel, et al., 2009; Putnam, 1995, 2000). Such 
connections can be conducive to more coherent policies because collaborative processes can 
promote shared knowledge, learning, mutual trust, and mutual motivation to address social 
problems. Sociologists have studied these kinds of connections, known as "social networks," to 
help understand the diffusion of innovation and knowledge since the end of the 19 th century. 
More recently, studies show that some organizations are more flexible than others; that is, 
they are able to bring to bear more varied, but relevant, expertise and knowledge on common 
problems through high-guality internal and external connections. These connections can form 
"communities of practice" that enable actors to more flexibly manage the freguently changing, 
social dimensions of research knowledge, as well as to contextualize it using the working 
knowledge of local actors (Barnes, et al., 2011; Brown & Duguid, 1991; Wenger, McDermott, & 
Snyder, 2002). 

This line of research and theory comports with newer more constructivist models of knowledge 
diffusion and innovation that cast research use not as a one-way transmission of research to 
the users, but instead as a "social process" (Honig & Coburn, 2008; Hood, 2002) involving 
social sense-making or interpretation of varied types of knowledge (Spillane, Reiser, & Reimer, 
2002; Weick, 1995). This search and incorporation process helps individuals and organizations 
integrate often de-contextualized research findings into their policies and practices (Argyris & 
Schon, 1996; Coburn & Russell, 2008; Coburn & Talbert, 2006; Kennedy, 1982). The process 
can even be generative, as learning, working, and innovating can interact such that members 
of communities of practice or professional knowledge networks actually construct new forms of 
"useable" knowledge for guiding action (Barnes, et al., 2010; Brown & Duguid, 1991; Lindblom 
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& Cohen, 1 979). While scholars have examined the socially constructed and determined nature 
of knowledge diffusion, acquisition, and use in organizations such as businesses, schools and 
school districts, they have not applied these same lenses to SEAs. Current studies have offered 
little information about how SEAs are organized to search for and use the research knowledge or 
advice that they employ in school improvement strategies. 

As SEAs are pushed to take on an increasing burden for improving low-achieving schools and 
districts, this chapter explores how the structure, strength, and qualities of social networks are 
related to the use of research or evidence in SEA school improvement strategies. To better 
understand the relationship between social networks and the process by which SEAs in this 
study incorporated knowledge into state school improvement strategies, we examine Chapter 
3's network findings in more depth. Using a subset of network data and our qualitative data, 
we analyzed the direct connections, or "ego networks," of the most influential, well-connected 
knowledge brokers, including their relationships with the most influential external organizations 
in the networks. As defined in Chapter 3, influential participants were those (with many in- 
ties) who were highly sought after for research and practitioner information; that is, they were 
mentioned as a source of advice by many SEA staff. Well-connected participants were those 
central actors (with many in-ties and out-ties) who both sought research ideas and information 
from a range of sources, and, at the same, provided information to a range of SEA colleagues, 
thus serving as knowledge brokers in the research or other networks. 1 By linking the survey 
network data for these individuals (in State A, offices or departments) to their interview data, 
we were able to create a more holistic view of the networks at the level of individuals' direct, 
interpersonal ties. We also examined the strength of in-ties as well as out-ties, and asked not 
only to whom respondents turned for knowledge, but also how they did so, and why. 

In this chapter we continue to address our study's first two research questions. More specifically, 
we consider how strong network structures emerged or were organized in the three states, what 
accounts for the stronger ties in these groups, and their role in SEA's search and incorporation 
process. We also explore how stronger network connections can support stronger knowledge 
use through influential SEA knowledge brokers' "core networks." 2 


1 Influential or prestigious actors and the most central, well-connected brokers (individuals or offices) 
tended to overlap considerably, as they often do in centrality analyses using different measures. 

2 We did not conduct core or cluster analyses here. Rather, to identify the most influential or the most 
well-connected actors we used measures of in-degree, out-degree, and betweenness centrality (see Chapter 3 for 
details), and then examined their direct connections using survey and qualitative data to identify what we have 
labeled as core networks. Our thinking about how these strong core networks develop, and their role in the search 
and incorporation process, has been influenced by Wenger, McDermott, and Snyder (2002) who have described 
similargroups in business settings. 
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Core Networks and the Incorporation 
Process 

The patterns we found in the survey analyses in Chapter 3 were in large part reflected in the 
smaller, more interpersonal ego-networks (direct ties) of influential individuals in both the 
research and practitioner networks for States B and C. While many cross-office and external 
network ties facilitated broad searches and the flow of diverse ideas from varied sources, even 
with relatively weak ties, a set of more influential, well-connected SEA brokers brought research 
and other kinds of information from these different network sources into smaller working groups 
with whom they interacted more frequently and who were influential in one another's work. 
Many of the strongest relationships were in these relatively small core networks whose members 
collectively addressed problems of school improvement. Core networks generally included 
leaders and other staff within school improvement offices, a few key external organizations and, 
in a few cases, colleagues from other departments in the agency. 

These networks enabled influential, well-connected SEA knowledge brokers, who were typically 
central in both the practitioneras well as research networks, to incorporate research into strategies 
that were workable in the context of their respective states. As our framing research has shown in 
district settings, we found that SEA staff in all three states searched for and incorporated research 
into their school improvement strategies through a distinctly social process in which the network 
members interpreted, challenged, and otherwise made sense of research along with other kinds 
of information over time (Honig & Coburn, 2008; Spillane, Reiser, & Reimer, 2002). The search 
and incorporation process was ongoing, and it took place over multiple group meetings that 
began prior to our study and continued through the final waves of the study. The incorporation 
process, as we illustrate below, was not necessarily orderly or systematic, but included a cycle of 
problem identification, development, implementation, feedback, and revision of improvement 
strategies or tools. 

In all phases of the search and incorporation process, social networks not only provided SEAs 
with access to multiple kinds of expertise and knowledge, but were also vehicles for coordinating 
components and developing more conceptual coherence in the school improvement system. 
Networks were also critical to the implementation, feedback and revision of strategies for 
continuous improvement. The core network groups used local practitioners' feedback, state 
professionals' experience, and external partners' knowledge of relevant research to contextualize 
various research findings in light of their states' school improvement needs. In contrast to 
models of research dissemination in which generalized, primarily decontextualized findings 
and products advanced by researchers are transmitted to users, in these core networks users, 
practitioners, and a few providers worked collectively to adapt research to address particular 
problems, and, in some cases, co-construct new useable knowledge for guiding action. 
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How and Why Strong Network 
Relationships Developed During Search 
and Incorporation: What Works and the 
Shared Problems of Practice 

We need this brain trust at the state level to continue pressing for what works: stealing 
ideas from other states, looking around, see what's working in our own system, looking 
at the literature. I mean, it's basically research-based best practices and action research 
best practices. (SEA knowledge broker, State B) 

The core networks or communities of practice emerged or were organized among influential 
externals, influential school improvement staff, and others in part because many SEA 
respondents valued research. But research was also highly sought after in forms that were 
practical and provided some guidance for action. SEAs often sought research that would address 
quite specific problems or needs. At the same time practitioner advice or expertise was highly 
prized, and thus strong ties developed with individuals and organizations that could provide, or 
assist in generating or adapting this mix of relevant, trusted information to state contexts. 

Across all three states, influential school improvement brokers formed strong core network 
connections with a few key external and internal sources who could assist them in matching 
or adapting reliable research and other evidence to specific school improvement problems. A 
director in State C, for example, recalled a request that was representative of many others in our 
states: "We're getting ready to implement the lead turnaround partner model with this group 
of schools. What kind of research do you have?" This request was directed to liaisons from the 
Center for Innovation and Improvement (Cll) who were influential members of core networks in 
both States B and C. But research was not simply transferred to an individual linking agent, or 
even to a team, in the case of this request and others similar to it. 

Instead, core networks of key school improvement staff and Cll staff worked collectively to 
incorporate research principles into workable state strategies with a common goal or problem 
in mind. This interactive, social process took place over multiple meetings in light of practical 
challenges and other kinds of evidence. A Cll liaison and member of a school improvement 
director's core network in State C described the development and adaptation process for a 
district guidance tool that became a key part of the improvement process in that state: 
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We developed it, but it was very joint in terms of, we first created it to solve a problem 
for a state... .[The school improvement director] said, 'our school improvement 
planning process is not working.' And then with them we devised the kind of solution 
and developed it... .So, it has been very co-developed. 

Thus during the incorporation process, core research networks frequently developed around 
states' school improvement problems and included an external source of expertise for research 
and development, working with an SEA staff team lead by a key school improvement broker. 

Influential SEA staff also formed strong ties with external sources who not only provided 
research, but also helped school improvement teams make sense of that research with practical, 
how to advice, through improvement models and work with colleagues in other states. One 
such staff member in State B explained his strong ties to the comprehensive assistance center 
(CAC) system this way: 

A [regional CAC], the Center for Innovation and Improvement, and the technical 
assistance agencies, those folks. That's where you go. They spend a lot of time doing the 
research, compiling it, analyzing it and so that's the first place I go. 

And then for strategies, for how to do it, first place I go is to . . . see what other states are 
doing and see what we can learn from other states. What's nice about [the regional CAC] 
is that they pull the states together on a regular basis so you can learn what the states 
are doing. ... If they figured outa wayto get itdone. 

This comment raises a classic problem identified in knowledge utilization studies over the past 
decades; that is, figuring out a way to translate research knowledge into less abstract, more 
concrete forms that can guide program development and implementation (Weiss et al., 2008; 
Argyris & Schon, 1974). The problem was embedded in the incorporation process and a few 
influential externals actors, as well as SEA knowledge brokers, created strong core network 
connections that enabled a collective approach to the problem. Moreover, influential externals, 
such as the regional CAC liaisons, also acted as "community developers," as well as knowledge 
brokers, during the SEA's search and incorporation process. As a staff person in State B described: 
"Our liaison says, 'I need to connect you with a guy in Alabama, or I need to connect you with a 
guy in California or Wisconsin or whatever.' So it's just a door into a set of resources." 

In State A, the well-connected school improvement director and a few staff formed weaker 
ties with multiple external providers, including the CAC system, as they addressed different 
components of their school improvement delivery system. For example, the SEA engaged its 
regional CAC and a national consulting firm to help in the design of its new regional assistance 
system. As an SEA manager described the process: 
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First of all we were working with our comprehensive center. They had been doing some 
national research about how assistance was being provided regionally in other states.... 

So the comprehensive centerwasone partner, but we hired up another organization 

[that] really became good partners with us in the thinking through how do you figure 
out what schools and districts need? . .They've done work around the country providing 
school support and district support systems and system development. And they've 
done a lot of review work. 

In contrast, key school improvement staff turned to university faculty and consultants to inform 
the design of a new superintendents induction program and a related school board governance 
training program. These experts had researched and written about strategic planning, 
human capital management, and school board relationships. Thus, the SEA in State A formed 
connections with external sources of expertise because those sources provided opportunities for 
problem-solving and experiential knowledge drawn from early implementers of school-based 
improvement strategies or regional support systems. But they did so with multiple externals, 
more intermittently or as needed, with fewer ties, thus making their connections weaker than 
those in the other two states, and the Cll far less central in the state's work. 

The State A School Improvement Department developed stronger ties to the agency's own 
Research Department and Accountability Office (the latter connection was influenced by 
state law), both of which compiled local data to evaluate specific components of state school 
improvement strategies and that could be used to address emerging problems. One influential 
explained: 

Our key partner in getting information about districts and using that data . . . comes 
from that [Research Department].... We've been spending a lot of time pouring through 
[data] -trying to figure out, grapple with how we can best intervene in districts. Looking 
at our past practices, what worked and hasn't worked according to the data. 

This core network was formed, in part, as a group of influential school improvement staff 
grappled with common challenges using the kind of local research evidence that would allow 
them to figure out what works in specific contexts. The comments also describe how collective 
sense-making was embedded in the incorporation process. 

The examples in this section show how and why SEAs strong core networks (even some weaker 
network ties) emerged or were organized as a group of key actors, often with different kinds 
of expertise and knowledge, came together to address common problems, challenges and 
goals. As Wenger, McDermott, and Snyder (2002) found in business settings, communities of 
practice were formed or emerged as "people address recurring sets of problems together" (p. 
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26). Influential school improvement staff in States B and C with strong ties to key liaisons in 
the federal CAC system, or even in State A with moderate ties, were able to learn about relevant 
research and how to apply it from developers of school improvement models, and to do so 
collectively with states who were puzzling over similar problems to their own. Not only could the 
SEA staff in these states access research tailored to particular school improvement policy needs 
they faced, but they could then see how it might be put into action from other SEAs who were 
early implementers. States A and, to some extent, State B also formed core research networks 
that included their agencies' research offices and accountability departments to provide local 
data that could help them identify and address common problems. 

Strong Ties to Practitioner Networks 
during the Search and Incorporation 
Process 

The core networks in all three states also included strong ties to networks of practitioners "on 
the ground" and in professional associations, as well as to the research networks. Practitioner 
networks provided advice and feedback on how research- based improvement strategies were 
working in the field, what needed clarifying, or what could be changed. As one State B influential 
put it: "It's getting that feedback on what will work and what won't." An influential in State C said 
of a group of field-based practitioners with strong ties in his network: "They're the ones that 
see it in action and they know where our course correction should be. They're the ones that can 
say, 'This is working great, this is not working.'" In State A, influential school improvement staff 
reported on the role of a superintendents network in the incorporation process this way: 

The function, among other things, of that network is for them to be able to inform our 
work. We often bring things to them to get feedback on it at different stages in the 
process. Is this a good idea? Should we do this? 

Thus, during the incorporation process, strong network structures emerged through regularly 
occurring, routine connections to practitioner networks, and helped school improvement brokers 
make sense of their improvement ideas in light of practical considerations. An influential in one 
state explained, "And we regularly met. ..with a group of district school improvement people 
and. ..we use them as a sounding board. ...We would discuss our ideas with them.'... .So, that 
was our regular structure." 

Key SEA respondents also formed strong ties with practitioners in other states through 
professional associations so they could exchange information about how research might be used 
in practice, or what others were actually doing to address common problems and challenges. A 
respondent in State B explained: 
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If I had to pick one thing that's been the most helpful for me in terms of my thinking, 
it's what research are my peers around the country drawing from that I could also utilize. 
What decisions they're making, how are they handling various policy challenges in 
their states? 

Core networks of influential SEA brokers emerged through a collective problem-solving process. 
They gave our states access to expertise from different sources of evidence and, in these cases 
just above, allowed SEAs to incorporate that evidence with highly valued practitioners' advice 
into their school improvement strategies. 

Flexible Core Networks Connecting 
SEA Staff with Practitioners Enabled 
Responsive, Dynamic Improvement 
Strategies 

SEA networks also allowed for an implementation, feedback, and revision loop that kept their 
state school improvement strategies dynamic, responsive, and open to changing contexts 
or emerging problems, as opposed to top down and static. In some cases, our states' SEA 
knowledge brokers were at the center of overlapping networks that not only included external 
sources of research and agency staff, but, in turn, extended across levels of the delivery system 
to include the actual implementers of state strategies. These corresponding network structures 
were sometimes used for more than a sounding board for SEAs school improvement designs 
and tools. 

In these examples, networks of practitioner-implementers and SEA staff identified problems 
collectively and jointly developed tools to address them through multiple meetings and 
interactions. SEA staff, together with district leaders in State A, for exam pie, developed and revised 
research informed toolkits they could use when addressing common and persistent problems 
across districts. One such problem was how to capture what was happening in classrooms. In 
discussing what began as the joint development of a district protocol for classroom learning 
walkthroughs, members of a core group of school improvement staff in State A described the 
evolution of their broader, more general incorporation process: 

What's the process for identifying those challenges? It's a mix of what we bring and 
what [district superintendents] see. . . .We see it's a common need for multiple districts 
to have guidance on getting in the classrooms. So rather than having 10 different 
districts do that, let's come up with something in conjunction with them. 
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Another member reported: 

You can look at them and you'll see research embedded in these pieces. But those 
toolkits helped us only to a point. Because we began to talk with districts, the ones that 
we've had a nice long relationship with and they would say to us . . . 'Why don't you 
come to key meetings and when things bubble up, that's how we'll all know [what we 
need].' 

Therefore: 

I think we've grown to a place where it's real time need, really fluid work. And that's 
the best of our work. When people [in districts] are saying, 'We're struggling with this.' 
And we might end up at the table saying, 'we're all struggling with the same thing. But 
we'll ... see how we can broaden this conversation or deepen it, and use a little of the 
expertise that resides at the state.' That's how we're operating most recently. 

This kind of very fluid search and incorporation process, including problem identification that 
was tightly coupled to practitioner-implementers, was enabled in part by flexible network 
structures in all three states. State B, for example, used a process and structure with local districts 
corresponding to their collective work and strong network ties to key externals. A deputy director 
and her staff worked with a regional CAC liaison and the CII's Academy of Pacesetting Districts. 
The deputy director then brought together her staff, representatives of a few area education 
agencies (AEAs), and staff in selected districts to set up "thought partnerships," using an 
incorporation cycle that corresponded to the one SEA staff had used with the CM. These core 
networks, including the district teams, met regularly and overtime to try out, make sense of, and 
revise a research-based district guidance model for improving districts' support systems in light 
of the districts' specific problems or needs. 

Networks, especially SEA brokers' core networks, allowed for a flexible interactive "top down- 
bottom up" learning loop that was open to the incorporation of new information and responsive 
to "real time" problems. In State C, webinar sessions created connections with practitioners, 
opened lines of ongoing communication, and were designed to address problems that arose 
as school improvement tools were implemented. For example, the director of the School 
Improvement Office and his staff regularly met with school practitioners in a webinar series that 
he described as an interactive problem-solving and learning process: "I'm going to have maybe 
four schools on the web conferences. We keep our web conferences very small, 1 2 to 1 6 people, 
so we can have a lot of interaction." He later continued: "That way, everybody is learning." 

Importantly, in all of the examples discussed above, influential external actors and key SEA 
brokers organized network structures that blurred traditional distinctions between problem 
identification, search, development, implementation, and feedback, as well as between the 
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concepts of knowledge producer and knowledge user. Not only did core network connections 
in these cases bring different perspectives into the development and implementation process, 
but they created what Lusi (1 997) and other researchers suggest are much more flexible and 
adaptable organizational structures for generating or managing knowledge so that it can be 
responsive to pressing school improvement problems. This is far different from what Lusi found 
when she studied her two SEAs. 

Indeed, flexible network structures enabled all of our states' school improvement strategies to 
remain open to contextual changes, new research, or promising practices, rather than becoming 
fixed and closed off from a cycle of improvement. As our examples showed, core network 
structures that included a few influential externals, such as the CAC system liaisons, helped SEAs 
manage the longstanding problem of accessing and matching frequently changing, sometimes 
contradictory, and often ambiguous, research findings to context-specific problems (Cohen & 
Garet 1 975; Lindblom & Cohen, 1 979; Weiss, et al., 2008). Moreover, the implementers or users 
had opportunities to try out new research-based improvement tools in the complex settings in 
which they worked, then revise them with SEA staff to take account of the contingencies that 
inevitably arise in district or school settings (Barnes, et al., 201 0). 

Core Networks and Shared Resources 
of Practice 

They've built that platform to align with the various components. Our coaches are 
invited to everything and so they're always on the web conferences. That keeps 
everybody in the loop. . .That's the other thing that I think the Center did. It gave us all a 
common language. Even here at the department, you need a common language. (SEA 
respondent, State C) 

As many of the examples in the prior sections show, members of the core networks worked 
together overtime to develop, refine, and adaptor use a set of ideas, protocols, processes, toolkits, 
documents and frameworks, many of which were informed by research. Thus, core network 
members developed and shared a "practice;" that is, a body of knowledge and resources they 
used to be more effective in their work (Wenger, McDermott, & Snyder, 2002; Brown & Duguid, 
1991). In some instances, research- based blueprints were adapted through core networks for 
use in the context of a particular state. An influential school improvement director in State C 
worked with a few improvement staff and the Cll liaison to adapt portions of the CII's Handbook 
on Restructuring and Substantial School Improvement (Walberg, 2007) for incorporation into a 
protocol for use in districts. The handbook contains a research-based checklist of indicators for 
schools, districts and teachers to use in identifying areas for improvement. He reported on this 
adaptation process of narrowing the indicator list within one core network: 
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When I saw the book, I just looked at it and said, 'This is exactly what we need. Here 
are the research-based indicators. Here's a blueprint for our schools that are in 
improvement....! think it's a little unwieldy' because there were so many.... That was 
[the Cll liaison's] role. He sat at those meetings where we [school improvement staff] 
discussed taking those indicators down to a narrower group. 

In State B, core network members, including key school improvement staff, partners from a state 
university, and a group of school improvement facilitators organized by a local professional 
association, worked collectively to adapt and incorporate a research-based protocol specifying 
Instructional Rounds (City, Elmore, Fiarman, & Teitel, 2009) into their statewide system of 
support. In State A, SEA staff, teams of practitioners, and several different external providers, 
collectively, though in different configurations, developed a set of tools for guiding district or 
school improvement processes: classroom learning walkthroughs as described earlier, a district 
data team toolkit, a self assessment for common planning, and more. 

In some instances shared resources of practice were refined over time through core networks 
and then used to design school improvement support systems.The SEA in State A, for example, 
engaged with sources of external expertise that included research providers and practitioners 
in the design of their regional school improvement system. This group drew on a set of district 
assistance processes that had been honed over several years through strong, long-standing 
network connections among key SEA school improvement brokers and the state's urban districts. 
An influential SEA staff reported: 

We'd had a set of activities that we had developed for those large urban districts, 

. . .assistance activities that have been honed over many years in collaboration with 
those districts. So as we [designers of new system] started to think about ... the services 
that we would initiate ... we drew from this set of resources. 

Finally, in a few cases, research-based models were actually developed by core network members 
for use in their state. State C worked with a regional CAC, a local college, and a state educational 
leadership group, to develop a multi-dimensional change model for school districts that would 
ensure these districts were not only identifying problems, but also providing schools with tools 
and conditions for improvement. According to an influential broker in the SEA: 

We worked with [the director of the regional CAC] and our partners at [local college], our 
partners at the [practitioner leadership group], and then various departments within the 
[SEA], And we reviewed research regarding districts' support to their lower performing 
schools. We also reviewed information regarding capacity building. ...Based on that 
information we. ..developed an operational framework, which was the change map. 
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Coherence 

All of these shared tools, guiding processes, and operational frameworks, together with the core 
networks themselves, supported the coordination of system components, and fostered more 
conceptual and technical coherence within the SEAs, as well as in their school improvement 
assistance systems. Most respondents perceived agency and system coherence to be "emerging" 
and "in progress" in the states, definitely not a "fait accompli." Still, staff most often discussed 
a set of key shared documents that embedded their improvement strategies and the tools or 
processes based on those documents, when asked about alignment, coordination, or coherence. 

In State B, the core network members discussed earlier developed a normative understanding 
and language for talking about an important principle in the state's school improvement 
strategy based on the classroom observation protocol they adapted and a shared interpretation 
of the City etal. (2009) book. This shared understanding and language eventually became more 
widespread in the SEA, as well as in the school improvement delivery system, through well- 
connected knowledge brokers, training sessions for instructional leaders, and various shared 
artifacts. The following articulation of this principle is drawn from researchers/theorists, such as 
Elmore and others (e.g. Cohen, Raudenbush, & Ball, 2003), and is one of many similar versions 
respondents used: 

If we're trying to improve student achievement, all pieces of the system have to be 
focused on the instructional core ... You can't make a difference unless you're focused 
on the interaction between the teacher and the student in the presence of content 
(Member of a core network in State B) 

When asked, "Do you think that the major components of your strategy are well-aligned or 
coordinated with each other?," a member of the School Improvement Office director's core 
network in State B reported: 

The department is ... bringing that effort into one place. And that's where I think the 
alignment is going to come from, because everybody gets on the same track so to 
speak. They're really working with the Elmore, Instructional Rounds. 

A large majority of respondents in this SEA understood and discussed, unprompted, Instructional 
Rounds during their interviews. 

This widespread, conceptual understanding of an important principle underlying the 
improvement strategy, the protocol for implementing it, and multiple documents describing 
it, constitute a set of shared resources of "practice" developed through core networks. These 
resources supported coherence across components of the delivery system and across multiple 
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offices within the School Improvement Department, and even across some other salient 
departments. During the incorporation process, the core network members focused State B's 
improvement strategy more specifically on classroom observations of instruction, student 
learning, and academic content. 

State B's School Improvement Framework (SIF), a research-based document described in more 
detail in Chapter 2, also provided a common organizing framework for accountability and 
school improvement delivery systems in the state. The SIF and tools that were derived from it 
were widely understood among SEA staff across departments involved in school improvement. 
Influential SEA staff and their partners used these resources to develop consistent messages 
and guides for the actions expected of schools or districts. Nearly every respondent discussed 
this framework or understood its key principles as a shared resource for coordinating action 
and developing system coherence. Three representative examples below are evidence of the 
broader pattern. 

A lot of our work is driven by the school improvement framework. You'll see data 
pieces in school improvement framework, you'll see professional learning, . . . you'll 
see standards for assessment and accountability. It's been built into the accreditation 
process. It's been built into field services work. . . We're writing into our grant criteria 
at every opportunity we have that [districts] ... have to meet the state standards, they 
have to use the school improvement framework, the planning process and all of that to 
inform that work (Office Director, State B). 

The school improvement framework, that's the one thing we all have a common bond 
with and understand and can help each other to support (School ImprovementAnalyst, 
State B). 

They [partners in the school improvement delivery system] have taken it upon 
themselves to align the terminology, the tools, the instruction, the feedback, the 
networking. . . They've based it all on the school improvement plan so there's a lot of 
alignment there (Federal Program Monitoring Director, State B). 

Key SEA actors, and their core network of partners charged with developing and implementing 
the system of support, derived a common theory of action from this state framework, then 
embedded that theory into a myriad rubrics, protocol or other tools and trainings to make their 
support components for low-achieving districts and schools more coherent. 

In State A, the state's theory of action for accountability and assistance was also embedded 
in a framework for coordinating action and policy that focuses on building district capacity. 
The district and school standards that comprise the framework drive the SEA's school and 
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district improvement planning, monitoring, and assistance work. The jointly developed set 
of improvement tools that were described earlier in this section are designed to orchestrate 
consistent action and support the use of a common set of research ideas across districts. These 
tools, which are embedded in the SEA's school and district self-assessment and improvement 
planning processes, also supported a shared understanding of improvement strategies among 
a wider range of SEA staff working on school improvement. 

State A also developed a collaborative network structure, connecting key school improvement 
actors within and outside the SEA, that is designed to build capacity in its improvement system, 
as well as orchestrate alignment across the different regions of that delivery system. Each of the 
Regional Assistance Team members is supervised by the School Improvement Department, but 
is also supported by an external partner who provides professional development and, through 
that, "new research and ideas" to each job type: the Regional Assistance Team directors, the 
math and literacy specialists, and the data specialists. The system's director reported: 

This [complex matrix design] creates an opportunity for alignment across all the regions, 
[and] building capacity for the people in those jobs, [by bringing] a whole host of new 
research and ideas from partners into the way these people carry out their work. 

The director, whose office was well-connected to other departments within the SEA, also meets 
with the regional teams and their partner organizations, which include SEA program offices, 
every other month to discuss common needs, and help the teams stay current on new initiatives 
emanating from the SEA. 

Brokers and core networks in State C likewise shared tools, protocols, and other artifacts of practice 
that could support more coherence in their SEA among those involved in school improvement 
and in their delivery system. For example, 

I do find that [the components of the school improvement strategy] are reinforcing 
each other and well-aligned. Teacher-leader training instruction draws heavily upon 
Wise Ways, which is embedded in the Indistar 8 planning. And the webinars that are 
provided to Strand I schools reinforce and support the Indistar ® planning tool, which 
heavily emphasizes data-based decision-making (State C manager). 

When asked, "Doyouseeanysetof common ideas or principles that underlie the core strategies?" 
one member of the School Improvement Office director's core network explained: "Well, 
definitely continual improvement. The whole system of Indistar ® planning, ...is based on the 
notion of continuous improvement." Similar to State B, not all, but many others we interviewed 
in State C had common conceptions of core principles underlying their improvement strategies, 
and a shared knowledge of key organizing tools of practice that embedded those strategies. 
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Finally, one network of the most influential broker in State C included ties to sources of 
knowledge or expertise from practitioner groups, university researchers, coaches working in 
the school improvement delivery system, and members of his own staff as a mechanism for 
examining, revising and coordinating activities across different improvement initiatives. One 
close contact to this broker said: "He pulls them together and has detailed conversation about 
tweaks and changes that we should consider moving forward. I would say that that team acts as 
the pivot point for a lot of the school improvement initiatives." 

Similar networks for coordinating across sources of information or school improvement 
components also existed in States A and B in different configurations. But the pivotal knowledge 
broker position was more distributed across multiple influential actors (or offices) in the other two 
states, unlike State C, where one person was central. Across all three states, network structures 
and the shared tools, frameworks or other resources that core groups developed, assisted SEAs 
in managing the ongoing challenges of building conceptual and technical cohesion into their 
improvement strategies or delivery systems. 

Core Networks and Social Capital: 
Collective Efficacy and Trust 

These kinds of highly interactive networks with strong ties among the members are associated 
with more of the relational qualities that Coleman (1988, 1990) and Putnam (1995, 2000) 
describe as "social capital." Social capital takes the form of mutual trust or collective efficacy, 
among many other forms, and can improve productivity and outcomes in organizations such 
as schools or business firms (Bryk & Schneider, 2002; Dutton & Heaphy, 2003; Goddard, 
2001; Goddard, Salloum, & Berebitsky, 2009). Survey data from measures of collective trust 
and efficacy, in combination with interview data, describe the extent to which respondents in 
this study perceived these positive qualities to exist in their relationships with network sources 
during the search and incorporation process, and especially how social capital functioned as 
knowledge brokers pulled knowledge from across these networks into core groups. 3 

Table 4.1 shows that, on average, respondents in States B and C perceived the levels of social 
capital in their direct networks for research, data, and practitioner advice to be quite high. Trust 


3 The quality of network connections was measured through a set of survey questions related to the level 
of trust and sense of collective efficacy that respondents reported among each group of individuals they named as 
key sources for a particular network (and in the case of trust, the information those individuals provided). Trust and 
collective efficacy measures were adapted from Bryk and Schneider (2002), Goddard (2001), Goddard, Hoy, and 
Woolfolk Hoy (2000), Goddard and LoGerfo (2007), and Goddard, Salloum, and Berebitsky (2009). See Appendix 
Table 1.3 for a list of survey questions and method for calculating the trust and efficacy measures. We did not in- 
clude these questions in the State Asurvey. 
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scores for example ranged from 5.4 to 5.6 on a six-point scale, suggesting respondents found 
their network sources to be trustworthy and able to provide valid information. Average collective 
efficacy scores are slightly lower, especially in State B, ranging from 4.8 to 4.9, and in State C's 
research network, with a score of 5 compared to trust in that network at 5.6. We also found 
more variation in efficacy scores than in trust scores, especially in the research and practitioner 
networks for State B. The collective efficacy measure captures respondents' views about their 
own capacity, together with that of their direct network members, to reach school improvement 
goals. This measure includes constraints that respondents and their direct network sources 
perceive, the resources they have, and the extent to which they believe the information they 
receive will actually help them meet their school improvement goals. 


Table 4.1. Social Capital in Knowledge Networks, States B and C 



State B 



State C 



Trust and Collective Efficacy 

Research 

Network 

Data 

Network 

Practitioner 

Network 

Research 

Network 

Data 

Network 

Practitioner 

Network 

Total respondents in each 
network 

121 

118 

115 

79 

79 

78 

Trust average 

5.5 

5.4 

5.4 

5.6 

5.6 

5.6 

Trust range 

2.0 

2.8 

2.5 

2.0 

2.5 

3.0 

Trust standard deviation 

0.6 

0.7 

0.7 

0.6 

0.6 

0.6 

Efficacy average 

4.9 

4.9 

4.8 

5.0 

5.1 

5.1 

Efficacy range 

4.0 

3.8 

3.8 

2.6 

2.8 

3.0 

Efficacy standard deviation 

0.9 

0.9 

1.0 

0.8 

0.8 

0.8 


Scale: 1 = strongly disagree; 2 = disagree; 3 = slightly disagree; 4 = slightly agree; 5 = agree; 
6 = strongly agree 


In State B, 14% of the respondents' efficacy scores in the research network and 18% of the 
scores in the practitioner network are lower than 4, showing they disagreed with the efficacy 
statements (Table 4.2). But in State C, only 9 % of the research network respondents disagreed 
with the efficacy statements, and 8% of the practitioner network members disagreed. Thus, a 
not insignificant proportion of respondents in State B perceived that research and practitioner 
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advice, as well as their network sources of it, would not lead to the desired outcomes or, perhaps, 
they felt constrained in what they could do. Interview data suggests that respondents in this state 
felt short-staffed and under pressure to respond to rapidly changing accountability measures 
and other mandates. 

Table 4-2. Number and Percent of Trust and Collective Efficacy 
Scores Below 4, States B and C 


State B 

Trust 

Efficacy 

Research 

0/93 (0%) 

13/93 (14%) 

Data 

2/98 (2%) 

12/98 (12%) 

Practitioner 

1/68 (1%) 

12/68 (18%) 

State C 

Trust 

Efficacy 

Research 

0/66 (0%) 

6/66 (9%) 

Data 

1/66 (2%) 

5/66 (8%) 

Practitioner 

1/52 (2%) 

4/52 (8%) 


Scale: 1 = strongly disagree; 2 = disagree; 3 = slightly disagree; 4 = slightly agree; 5 = agree; 
6 = strongly agree. 

But most influential SEA respondents we interviewed across the states perceived information, 
decision-making, evolving improvement strategies and ideas to be more trustworthy and 
efficacious within the context of strong network relationships and collective work. 4 For example, 
respondents reported an enhanced sense of efficacy from collective thinking and group work. 
When asked if the people in his core work network had the expertise to find and then use 
evidence to successfully improve low-performing schools, an office director in State B reported: 

There is no one individual that holds all the information, which is why we have a group... 
all of those different people hold enough pieces that we can have conversations and 
share information across the table that can. ..push us along... to that ideal goal at the 
end. 

Thus, resources such as communal memory and distributed expertise created a collective sense 
of efficacy and supported core network members during the search and incorporation process, 
such that no one person had to know everything. 


4 Recall that here we are using patterns in the qualitative data from a select sample of the more central, 

influential and/or well-connected SEA actors, while the survey tables are from a broader sample of everyone in the 
networks. 
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Indeed, while these social resources and trust in information sometimes developed out of the 
interactive incorporation process over time, some groups were formed to intentionally take 
advantage of a collective efficacy surpassing the capacity that any one member in the network 
would have. A knowledge broker in State C said, for example: 

It really starts with people who come together for the same pu rpose. They like this idea 
of improving schools, and they're good at it. We all have our own expertise, and we 
bring that to the table.... You don't build your team with all the same kind of players. You 
build your team with different areas of experience . . . outside and inside [the agency]. 

In State A, a respondent explained: 

The first person I pulled in was the accountability person. . . Then I pulled in the 
evaluation person ...from our Research and Evaluation Unit because. . . I'm the first one 
to say that... there's too much I don't know... .Then the three of us looked at what data 
that we thought we would have to collect. 

Research in other settings shows that more efficacious organizations are able to not only defer 
to varied kinds of expertise, but also take advantage of its shifting locations (Weick & Sutcliffe, 
2001). Core networks allowed some SEA staff to tap into such expertise within and across 
the boundaries of the SEAs, and to benefit from the accompanying social capital during the 
incorporation process. 

Collective trust in information and the expertise of others was also present during the research 
incorporation process, thus enabling core group members to "share ideas, expose ignorance, 
ask difficult questions and listen carefully... "(Wenger, McDermott, & Snyder, 2002, p. 27). When 
asked if and why she trusted research and other school improvement information exchanged 
within a core network, one influential office director in State B said: "Because we digest it 
together. And people challenge each other." She continued with an example of a similar process: 
"We solve problems. What are we going to do about this?. . ..And people bring in research and 
we'll table things and [then] come back to them with the research, and then we'll challenge 
the research." An influential leader in State C said: "I want to validate what [researchers are] 
saying. When you have those strong networks, you build upon that professional knowledge and 
practice... .We're not afraid to tell people we don't know something, either." 

Strongly connected social networks are more likely to generate or possess positive working 
beliefs and behaviors than traditional forms of control over adult work, such as hierarchical, 
bureaucratic authority. In the three SEAs, we found that social capital existed within the stronger 
network relationships, but also functioned in multiple ways during the search and incorporation 
process. Our examples provide insights into how SEAs were taking advantage of distributed 
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expertise, warranting the use of research and incorporating it into their problem-solving, not 
directly, but through a process of social sense-making within strong core networks. Relational 
trust allowed network members to ask challenging questions of one another, share knowledge, 
exchange ideas, and expose what they did not know, as well as what they did. 

Summary and Conclusions 

Cultivating professional knowledge networks in configurations that cross SEA boundaries may 
be an important means to not only spread innovation and increase the influence of research 
in SEAs, but also to develop expertise, collective capacity and, we found, to tailor research- 
based solutions to different local contexts. Some strong flexible network structures in all 
three states enabled a very fluid search and incorporation process that was tightly coupled to 
practitioner-implementers, external sources of research-development expertise, and, in two of 
our states, colleagues across state agencies in similar roles. One state relied more heavily on its 
more extensive internal research capacity when seeking research and development expertise. 
SEA members charged with developing, implementing, and improving school improvement 
strategies searched for and incorporated research, the advice of practitioners, and a range of 
other kinds of evidence into their school improvement strategies, not directly, but through a 
process in which the network members challenged, interpreted, or otherwise made sense of 
research in the company of others. 

The smaller ego-networks (or direct ties) of influential SEA knowledge brokers reflected many of 
the same patterns we found in our survey analyses of the full networks. A small set of influential 
knowledge brokers (or offices) were in the thick of the information flow with lateral cross- 
department and external networks giving them access to a broad set of knowledge and advice 
(even with weak ties). At the same time, these key SEA actors were forming smaller core networks 
with stronger ties to some key externals, within department colleagues, and, in some cases, a 
few cross-department colleagues, to incorporate research into state strategies. 

Several factors influenced why and how the network structures developed, and their role 
in SEAs search and incorporation process. Federal incentives, such as NCLB mandates, grant 
applications, or funding guidelines, played an important role in forming network structures 
and in creating states' demand for the advice, products and services of external intermediaries. 
But only a few of these external groups and the research they marshaled became influential in 
the core networks of key SEA actors, and thus in shaping potential solutions to states' school 
improvement problems. 

The strong core network structures developed among these influential externals, influential 
school improvement staff and others, in part, because SEA knowledge brokers wanted research 
in forms that fit practical problems and provided guidance for how to apply research principles 
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to strategy development and implementation. They often searched for research that would 
address specific problems or needs, for example, and for practitioners' advice about what was 
working from their view either on the ground or in similar and related agency roles. Strong 
network ties developed with external sources, such as the federal CAC system, that could help 
SEAs generate or adapt this mix of relevant, trusted information to state contexts. The CACs, 
for example, because they were networked into the federal policymaking cycle, could help 
SEAs identify timely, relevant research or potential research-based solutions to problems that 
challenged states as they attempted to respond to federal policy pressures. In doing so, they 
could assist states in filtering out a large volume of sometimes inconsistent or irrelevant research. 

Thus, SEAs strong core networks (even some weaker networks) developed as a group of key 
actors, often with different kinds of expertise and knowledge, came together to address common 
problems, challenges and goals, typically embedded in specific state contexts. Social networks, 
especially stronger core networks, enabled SEAs to manage three longstanding problems that 
have been obstacles to the use of research in many organizations: matching research to pressing 
school improvement or other social problems; translating research into guides for program 
development and implementation-tools for guiding relatively consistent, intentional action; 
and, finally, addressing research overload, more specifically creating the collective capacity 
needed to sift through and interpret an ever-growing body of research. 

Some strong network structures connecting SEA knowledge brokers to practitioners were used 
for more than feedback for SEA's school improvement designs and tools. Afew SEA brokers were 
pivotal in overlapping networks that connected them laterally to important internal and external 
expertise, but also connected them vertically across the system to the implementers of school 
improvement designs. These overlapping, corresponding networks extended the incorporation 
process (collective problem identification, development, implementation and revision, not 
necessarily in this order) to practitioners who were attempting to incorporate SEAs research- 
based improvement designs into their daily practice. 

Importantly, overlapping and strong flexible networks allowed for an implementation, feedback 
and revision loop that kept their state school improvement strategies dynamic, responsive, and 
open to changing contexts, emerging problems, and even new research, as opposed to top 
down and static. Influential sets of externals and key SEA brokers organized flexible network 
structures that obfuscated traditional distinctions between problem identification, search, 
development, implementation, and feedback, as well as between the concepts of knowledge 
producer and knowledge user. 

Members of core networks worked together over time to develop, refine or adapt a set of ideas, 
a theory of action and the guidance rubrics, frameworks, protocol or other tools that embedded 
these ideas and theories. This body of shared knowledge and resources was based on research 
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principles, among otherfoundational sources and commonly understood by network members. 
The members shared understanding and theory of action eventually became more widespread 
within the SEAs and inschool improvementsupport systems through well-connected knowledge 
brokers, training sessions for teachers and education leaders, and various shared artifacts. 
Influential school improvement staff also used frameworks and other tools to orchestrate more 
consistent guidance for districts and schools, and to coordinate across components of the states' 
school improvement delivery systems, as well as across a few departments involved in school 
improvement within the SEA. 

Social networks, but especially the strong core networks of key school improvement staff, 
supported more technical alignment and conceptual coherence in states' improvement delivery 
systems and in the agencies. In all three states however, respondents discussed coherence as 
emerging or a "work in progress" consistent with the idea that organizations "craft coherence" in 
an ongoing process rather than ever fully accomplishing it in the crowded, multi-level education 
system (Honig & Hatch, 2004). 

Through a collective problem-solving process, networks enabled our states to access important 
evidence from different sources of expertise, and benefit from the social capital in the stronger 
core network relationships. These core networks were sometimes designed to capture the 
collective efficacy found in distributed expertise, and members drew relational trust inherent in 
well-established relationships. But members also developed a mutual trust in information and 
expertise by collectively challenging or otherwise validating information using different causal 
frames for "what works," from observations of local practitioners, to local data and research 
findings. 

Research has shown that flexible organizations are more efficacious than many because rather 
than drawing solely on traditional lines of bureaucratic authority, they defer to varied sources of 
expertise, and follow that expertise to shifting locations inside or outside their boundaries. We 
are not suggesting that the SEA organizations we studied have been completely transformed 
from rule-bound, compliance organizations as their responsibilities are in part constructed by 
federal and state rules. But in terms of their responsibility for improving districts and schools, 
network structures in all three of the states enabled a fluid search and incorporation process and 
a "lived" organization more flexible and less segmented than earlier organizational researchers 
have found. 
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CHAPTER 5 

External Organizations 

Lead Author: Diane Massell 


In this chapter, we explore in greater depth the SEAs relationship with external organizations 
and groups that provided research and other types of knowledge about school improvement. 
Understanding the extent to which, where, and why SEA staff turn to external intermediaries 
is important for those seeking to provide such information to SEAs, and for those SEA leaders 
seeking to expand or refine their staff's involvement with outside groups. Such connections 
can help SEAs innovate by gaining them access to specialized knowledge (Sorensen & Torfing, 
2012) and improving the fit of policy to the urgent needs of educators (Lusi, 1 997). Research at 
the school district level has shown that variation in the availability of, or districts' engagement 
with, external intermediaries can impact their ability to effectively address problems of practice 
(Firestone, Rosenblum, Bader, & Massell, 1991; Hannaway & Kimball, 2001; Massell & Goertz, 
2002). Beyond its utilitarian value, Rowan (2002) provides a compelling neo-institutional 
argument that the dynamics of education reform are driven in part by the structure and 
composition of the organizational populations in the environment of U.S. schools. He notes 
that there has been a vast expansion in the number and variety of external organizations in the 
business of school improvement, providing information, training, materials, and programs. We 
seek to add to that understanding with a descriptive look at the contours of these organizations 
at the level of the SEA in the same school improvement domain, and to explain or propose 
questions about the patterns that we find there. 

We begin by exploring the extent to which SEAs turn to external sources for information on 
school improvement. We then describe the types of external sources included in these searches. 
Finally, we discuss factors that appear to explain why SEA staff in the three states turned to these 
sources, including the qualities of the external partners. 
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The Scope of External Search 

As discussed in Chapter 3, the vast majority of SEA staff in our study turned exclusively or 
primarily to their own colleagues for knowledge about school improvement. But a subset of 
staff in each of the three states also sought research or other types of information from sources 
outside their own agencies. Specifically, between 24% and 32% of our survey respondents were 
"boundary spanners," 1 that is, individuals who searched outside their agencies for research, 
data, or practitioner advice (Table 5.1 ). 2 By contrast, two-thirds or more of SEA staff identified 
internal sources for similar information. 


While the overall numbers of boundary spanners are fairly low, they belie the influence that 
external organizations played in the knowledge accessed and used by SEA policymakers, 
particularly by the staff most directly involved in assistance supports for low-performing schools 
or districts. Indeed, roughly half of all staff involved in school improvement work to some extent 
agreed that external organizations provided the best and most useful information resources for 
their work (Table 5.2). 

Furthermore, in almost every knowledge network across the states, SEA staff rated their 
strength of ties to external sources as either similar to or stronger, on average, than their ties 
to SEA colleagues across departmental silos. 3 In other words, respondents indicated that 
communications with external organizations occurred about as often or more often than with 
colleagues in other departments in their agency, and they rated the advice they received from 
externals as similar to or more influential than these colleagues. (See AppendixTable 5.1 .) 

The group of SEA staff members who sought information outside of their agencies collectively 
interacted with a large number of distinct sources of information across their research, data, 
and practitioner advice networks. 4 Altogether, State A survey respondents named 58 different 
external sources, State B identified 74, and State C identified 50 sources (Table 5.3). In every case, 
the external research networks were the largest, ranging from 37 to 42 distinct organizations, 
followed by the practitioner network (21 to 39), and the data networks (12 to 24). These numbers 
suggest that SEAs are drawing from a broad pool of external expertise and information about 
school improvement. 

1 The term "boundary spanners" is used in multiple literatures. Here, we draw loosely on the terms de- 
fined by the organizational theorist W. Richard Scott. Boundary spanners are those who act as a "bridge between an 
organization and their [knowledge] exchange partners, competitors and regulators" (Scott, 1998, p. 196). 

2 Our interview data suggest that these figures slightly under-represented the full scope of staff interac- 
tion with external sources. For instance, a few SEA respondents identified external sources of information during 
the interviews, but did not report any sources on the survey. 

3 The one exception to this pattern was in State C's practitioner network. 

4 This count of "distinct” organizations does not include "generic" mentions, such as "universities" or "con- 
sultants." 
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Table 5.1 Percentage of Respondents Naming External Sources for 
Research, Data, and Practitioner Advice, by Network and State 



State A 

State B 

State C 

Research 

24% 

29% 

32% 

Data 

9% 

17% 

15% 

Practitioner 

25% 

29% 

17% 


Table 5.2: Usefulness of External Resources in Work, by State (% of 

survey respondents) 

Rate the extent to which you agree with the statement: "My best resources for information 
useful in my work are located outside the state department of education." 



State A 

State B 

State C 

Strongly Agree 

9.1% 

5.4% 

8.1% 

Agree 

21.8% 

23.8% 

19.4% 

Slightly Agree 

26.4% 

20.2% 

19.4% 

Slightly Disagree 

12.7% 

19.6% 

13.1% 

Disagree 

24.5% 

23.8% 

28.8% 

Strongly Disagree 

4.6% 

7.1% 

11.2% 


Table 5.3: Number of Distinct External Sources, by Network and 

State 



State A 

State B 

State C 

Research 

37 

42 

41 

Data 

12 

24 

19 

Practitioner 

22 

39 

21 

Total Number* 

58 

74 

50 


*Total number in each state is an unduplicated count of external sources across the three 
networks. 
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These SEA staff reached out to an average of 2.4 external sources in the research network and 
2.0 in the data and practitioner networks. 5 About 30% of the survey respondents who named 
external sources listed just one such source, and another 30% mentioned between 5 and 13 
external sources. Based on our interviews, we anticipated that the SEA staff making the most 
frequent (five or more) connections to external sources would be in leadership positions in key 
school improvement offices. Our survey confirmed that managers directly in charge of the school 
improvement work had among the largest external networks, and, as discussed in Chapters 
3 and 4, were key knowledge brokers to whom many internal staff turned for information, 
especially in States B and C. In State C, for example, the head of the School Improvement 
Office played a critical role in identifying and soliciting the knowledge and support of external 
intermediaries, and internal staff throughout the agency, in turn, relied heavily on this leader 
for school improvement search and guidance. But in each state, our survey also revealed that 
boundary spanners came from all tiers of their respective organizations and were as likely 
to be 'regular' office workers as division heads, or executive-level leaders. There were some 
differences across the states, with a larger share of leaders in States A and B mentioning higher 
numbers of external sources than those in State C. 6 

While most of the SEA staff who reached out to external sources reported connections with 
multiple organizations, a sizable share of the external organizations themselves were named 
by just one SEA staff member on the survey, suggesting that many sources tended to be very 
particular to individuals. In State A, two-thirds of the distinct outside organizations or groups 
were named by just one SEA staff member; in States B and C, about half of the external sources 
were also identified by only one person (54 and 50%, respectively). 

The fact that so many of these organizations were named by j ust one person suggests that these 
relationships were very particular to the people involved, and that one SEA staff member served 
as the main contact for the work and the knowledge exchange. These singular connections 
may come about from person-specific experiences, such as an individual's prior relationship 
with a graduate faculty advisor, or a previous work colleague. The research director in State 
A, for example, pulled in several of her former graduate school faculty to give presentations 
at the agency. Singular connections could also arise when external organizations or groups 
provided information very specifically targeted to the roles and tasks of the SEA staff member. 
State B had an unusually large number of state-level professional membership organizations 
that were organized around job functions, where such individual linkages arose. In another 
example, one SEA director served on the governing board of a national assessment consortium, 


5 SEA staff in States B and C, where we collected individual-level data, reached out to an average of three 
external individuals. 

6 In State A, 4 of the 10 respondents (40%) mentioning 5 to 13 sources were heads of offices or depart- 
ments, as were 9 of the 14 (64%) in State B, and 2 of the 9(22%) in State C. 
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and had formed particular connections not named by others in his SEA. Finally, individual SEA 
staff members also identified federal program officers or representatives in their own respective 
categorical programs, connections that were often unigue. 

Froman organizational management perspectives preponderance ofindependent connections 
between SEA staff members and external organizations indicates potential vulnerabilities in the 
external-internal exchange of information. If that external source provides knowledge that is 
essential to the functioning of the SEA, and the single relationship is with an individual who 
is not internally well-connected (a knowledge broker), or with people who are not likely to be 
in the SEA for a long period, the knowledge may not be distributed, or the exchange may be 
broken and disrupt the work. In contrast, when two senior directors left the State B SEA, a broad 
set of relationships between the SEA and the external providers helped to ensure that the work 
they were engaged in continued. Staff from a regional comprehensive assistance center (CAC) 
also facilitated this continuity. 

Types of External Organizations 

Staff in the three SEAs turned to a range of different types of external organizations or groups 
for information. We categorized these sources into six types, along with an "other" and an 
"unknown" category, using the following definitions: 


» Federal government. This category encompasses federal agencies, 
including several offices within the U.S. Department of Education, and 
federally funded centers, such as the ESEA comprehensive assistance centers, 
regional education labs (REL), and other technical assistance centers that 
support the implementation of more targeted educational programs (e.g., 
special education or vocational education). It also includes state- or federally 
authorized and funded centers located in a university or research organization. 

» State, regional, and local government. Regional agencies include 
organizational units within a state, such as intermediate education units, 
and regional organizations that cross state boundaries. Local agencies and 
organizations include local school districts and cross-district networks of 
educators. 

» Professional membership associations (PMA). These include all 
organizations that are funded and governed by and on behalf of their 
professional members. Examples include subject-matter associations and 
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associations representing specific job functions at both the state (e.g. state 
association of mathematics teachers, district superintendents association) and 
national (e.g., National Council of Teachers of Mathematics, Council of Chief 
State School Officers) levels. 

» Institutions of higher education (IHE). IHEs include universities, colleges 
or community colleges. Research centers at IHEs with multiple sources of 
funding (and not solely or principally chartered and funded by government) 
are classified in this sector. 

» Research organizations. These are independent organizations such as RAND 
or the Consortium for Policy Research in Education that operate primarily to 
conduct research studies. Some organizations in this category may provide 
technical assistance, or develop and sell programs, but research is their major 
function. 

» Provider organizations. Independent provider organizations primarily 
develop and sell programs or services. These would include accrediting 
agencies, for example, or testing organizations. They may conduct research, 
but that is a smaller part of their operations. We did not distinguish between 
non- or for-profit provider organizations. 

» Other. This category includes the few identified organizations whose primary 
focus is advocacy, as well as foundations. 

» Unknown. Respondents occasionally named unspecified generic sources 
(e.g., "administrators"), or acronyms that referred to various organizations we 
could not identify. 7 


The analyses presented below are based on the number of times survey respondents in each 
state reported a connection, or tie, to sources in each knowledge network. Thus, for example, 
respondents in State A's research network mentioned turning to the 37 distinct external sources 
a total of 59 times. As reported below, one-third of these 59 mentions were federal government 
organizations. 


7 Generic types that were specific enough, such as "universities" or "district administrators," were classified 

with their respective types, in these cases, IHEs and State/regional/local government. 
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Research knowledge networks 

Looking acrossthe three states, we f i n d that S EAstaff relied most heavily on thefederal government 
and various professional membership associations for research on school improvement (Figure 
5.1 and AppendixTable 5.2). Federal government organizations received 32% (State A) to 42% 
(State C) of external mentions. Professional membership associations (PMAs) were mentioned 
about one-third of the time in States B (33%) and C (31%), but only 10% of the time in State 
A. Although fewer SEA staff sought research from institutions of higher education or research 
organizations, one to two universities did play an integral role in school improvement designs 
and delivery in States B and C, and SEA staff in State A often turned to university faculty and 
research and provider organizations for help on specific projects. SEA staff in State A were also 
somewhat more likely to turn to foundations or advocacy organizations for information. 

Figure 5.1: External Sources for Research Knowledge (Percentage of 

Total External Mentions) 


Govt: Federal 


Govt: State/regional/local 


Professional Membership 


Higher Education 


Research/Provider (NGO) 


Education Journals 


Other/Unknown 
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Figure 5.2 (and Appendix Table 5.3) break out the different sources of information within 
the federal government sector. Across the states, federal agencies or centers were the most 
freguently mentioned source, particularly the U.S. Department of Education and its divisions, 
such as the Institute for Education Sciences, the Office of Special Education Programs, the 
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National Center for Education Statistics, and the What Works Clearinghouse, among others. The 
states pulled information from different types of federal centers as well. State A staff mentioned 
more and different RELs in their networks, while State B and C were more likely to identify and, 
in interviews, talk about their work with several of the federal CACs. 

Figure 5.2: Federal Government Sources for Research Knowledge 
(Percentage of Federal Government Mentions) 
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SEA staff sought research from a range of PMAs as well. Many of the PMAs were organized 
around specific subject matter content, such as state and national reading, mathematics and 
technology organizations, or teaching and learning more generally, such as ASCD. A relatively 
large number of occupationally-focused PMAs were cited, such as the Council of Chief State 
School Officers (CCSSO), state-level PMAs representing superintendents, principals and federal 
program administrators, and regional organizations, like the Southern Regional Education 
Board. The three SEAs, however, reached out primarily to national, not state, PMAs for research 
advice. National PMAs represented 85% of total PMAs mentions in the research network, ranging 
from 100% in State A to 77% in State B. Two-thirds of these national PMAs were occupationally 
focused, while one-third were content-focused (AppendixTable 5.4). 
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Practitioner network 

SEA staff turned to a different mix of external sources in the practitioner network (Figure 5.3). 8 
Staff in State A turned most often to school districts and district networks (33% of mentions), 
while districts were not mentioned at all in State C and only 2% of the time in State B. Area 
education agencies (AEAs), a layer of government institutions absent in States A and C, received 
16% of mentions in State B. States B and C relied most heavily on PMAs: PMAs received 43% of 
mentions in State B and 47% in State C, but only 7% of mentions in State A. In contrast to the 
research network, SEA staff reached out primarily to state-based PMAs for practitioner advice. 
State-based PMAs represented 70% of total PMAs mentions in the practitioner network across 
the three states (although they ranged from 33% in State A to 86% in State B). Nearly 80% of the 
PMAs in the practitioner network were occupationally-focused organizations (Appendix Table 
5.4). 

Figure 5.3: External Sources for Practitioner Knowledge (Percentage 

of Total External Mentions) 
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8 We did not include the data network in this section because of the relatively small percent of SEA staff 

seeking advice from external organizations in this network. Information on the data network can be found in Ap- 
pendixTable 5.2. 
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Explaining Variation 

As the preceding section showed, the external school improvement advice networks of these 
SEAs shared many similarities, but also substantial differences. As discussed in Chapters 3 and 
4, some external organizations showed up as more influential and central in the knowledge 
networks of States B and C than in State A, with the federal CACs being a primary example. In 
States B and C, these federal centers frequently provided research and other supports to SEA 
staff as they reviewed their approach to school improvement, while the CACs received only brief 
mention in State A. The three SEAs also varied in the extent to which they turned to PMAs and 
local education entities for research or practitioner advice. 

The explanations for such differences in these external environments revolve around a complex 
interplay of factors, including the internal staff capacity of the three SEAs, the stage and focus of 
their school improvement work, and the structure of intermediaries in the states' environments. 
Personal relationships, or a history of institutional connections, reputations, and the availability 
or visibility of external sources also played a role in which specific external organizations were 
brought into the SEAs. Here we describe in more detail some of the differences that emerged 
across the states, and why. 

SEA capacity to address the challenges of 
school improvement: The role of the CACs 

In 2002, federal CACs were redesigned with a mandate to build SEA capacity to implement 
NCLB with research-based policies, strategies and programs. This emphasis on research grew 
out of criticisms that previous federal technical assistance supports lacked rigor, and were not 
sufficiently grounded in evidence (Policy Studies Associates, 2000). The new design became 
operational in 2005-06, and created a two-tiered system of 1 6 regional and five national content 
centers. The regional CACs were to provide direct technical assistance to the SEAs, using the 
research collected and synthesized by five national content centers on NCLB priority topics, such 
as high school reform, assessment, and teacher quality. One national content center, the Center 
on Innovation and Improvement (Cll), focused specifically on school restructuring and statewide 
systems of support (SSOS) for school improvement. In general, the national content centers 
were to work with the regional CAC liaisons, which would then, in effect, "push in" information 
relevant to needs identified in each SEA's annual technical assistance plans. In addition, the 
national content centers hosted annual meetings for SEA staff and their regional partners to 
expose them to the latest research in their areas. Over time, national content center staff played 
a more direct role in working with many SEAs on specific problems of practice (Massed et al., 
2010 ). 
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Although the CACs came with attractive financial resources, as well as a high- level of expertise, 
SEA engagement with them varied (see Massell et al., 2010). Our states suggest that whether 
and how states seek out and use these resources may reflect differences in their own staffing and 
internal capacity, and, in this case, the needs that they had for school improvement design when 
the new CAC system was being introduced in the policy environment. 

While SEA staff in State B were highly skilled-many of their leaders and mid-level managers 
had doctorates-they perceived themselves to be extremely understaffed to carry out many of 
the demands being required of them. To respond to the NCLB mandate to create a SSOS, they 
turned to several intermediary organizations, including a university, AEAs, and an interstate 
accreditation association, to design and provide its major components. Similarly, internal 
staffing constraints led them to request that their regional CAC house a staff member on site 
to help them follow through on work plans they had negotiated. This unusual request was 
granted, and the regional staff member became a regular presence in the agency. Among other 
things, she helped facilitate monthly meetings of a core team of internal and external players in 
the SSOS delivery system. In these conversations, as questions arose about the coherence and 
effectiveness of the work, the regional CAC was present and available to offer research from and 
develop relationships with the national content centers. Through this avenue, the Cll began to 
help review State B's SSOS. Among other things, the Cll advised them in designing a principal 
survey and a more comprehensive evaluation of SSOS coherence and influence on student 
outcomes. State B participated in the Cll Academy for Pacesetting States to exchange ideas with 
other state teams and develop specific school improvement work products; districts from State 
B later joined CII's Academy for Pacesetting Districts for similar purposes. 

State C had even more severe staffing constraints than B, with only one research staff in the 
entire SEA and few people in the school improvement area with advanced degrees or relevant 
experience in the field (see Chapter 2). State C was also at a point of reconsidering its supports 
for low-performing schools and districts. One major problem of practice for them was how to 
get sites to produce meaningful school improvement plans, rather than building plans based 
on their own preferences or ideas rather than evidence and research. Furthermore, some sites 
were reintroducing the same documents year after yea pan indication thattheywerejusta paper 
exercise to some, and not effective levers for change. 

As they were wrestling with these questions, the director of school improvement attended 
one of the annual conferences hosted by the Cll for SEA staff and their regional CAC liaisons. 
When he heard presentations from CII's Handbook on Restructuring and Substantial School 
Improvement (Walberg, 2007), he believed it provided a solid, research-based answer to this 
problem of practice. 
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The [Handbook on Restructuring and] Substantial School Improvement, that's what we 
based everything on. I read that book, I loved it and I called and said, 'This is what I want 
in [our state]. I've got to have this kind of support.' 

State C invited the Cll to the state and, together with other external partners and SEA staff, 
including a PMA foundation and coaches in their delivery system, tailored the indicators to their 
own specific needs. This was the beginning of what would prove to be frequent and on-going 
interactions with the Cll to adapt or seek out new research and expert opinion. Their collaboration 
also resulted in the development of a web-based platform for the indicators.The Handbook, and 
Cl I's Mega System: A Handbook for Continuous Improvement within a Community of the School 
(Redding, 2006), became required reading for SEA staff and providers. And, like State B, State C 
was a member of the Cl I's Academy for Pacesetting States. 

In contrast, the basic components of State A's school improvement system had been in place 
for several years; the over-arching design was not a target for major review. As a result, State 
A was more likely to "pull in" advice and support from a range of external partners on an as- 
needed basis, to co-develop discrete and specific tools and resources to address problems 
that SEA staff identified through their work with local school districts. External partners were 
identified and selected for their expertise in a particular area. For example, State A worked with 
the National Institute for School Leadership (NISL) to develop guidance to schools and school 
districts on creating professional learning communities, and with different national consulting 
organizations to develop a toolkit to support districts in data-driven decision-making within a 
cycle of inquiry, and guidance on common planning time and school district reform. The SEA 
turned to nationally-recognized scholars to assist in the design of new leadership training 
programs for school district superintendents and school board members. A REL reviewed 
research underlying State A's revised SIF. Still other organizations provide on-going research 
and support to members of State A's regional assistance teams. As a member of the School 
Improvement Department explained: "This ensures we've got people with expertise in [each] 
area overseeing the job, ...and [brings] a whole host of new research and ideas. ..into the way 
that the [regional assistance team] people carry out their work." 

The search process was facilitated by the SEA's Research Department, which provides research, 
analysis, and planning and evaluation support to departments across the agency. As one SEA 
staff person noted: 

We have over the last couple of years created the Research Department... .Really smart, 
really good people. We often turn to them and say, 'What's out there around this?' And 
they are very good at being able to access the information that might be brought to 
bear in this problem-solving. 
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The Research Department assists other SEA departments in the design of program evaluations, 
selection of evaluators, and monitoring of evaluators' work. It also vets the qualifications of 
external providers for both the SEA and school districts. 

Intermediary environments in the states 

At the outset of this study, we sampled SEAs that on some measures appeared to vary in the 
extent to which they had access to resource-rich organizations in their own state intermediary 
environment. We reasoned that the availability and potential familiarity of local, in-state 
organizations would impact patterns of search and use. Under our initial assumptions, State 
A was highest on the scale of in-state intermediaries since it was populated by numerous and 
prominent research and academic institutions. Although State B had an extant set of AEAs, 
funding for these organizations was in decline, and we knew from other studies that several 
were resistant to or unengaged in school improvement work. National observers involved in 
State C reported that it had few intermediaries, and no plans to develop or fund any new in-state 
institutions or centers to deliver services. 

Identifying all potential external sources in each state was beyond the scope of this study. 
Rather, we use the number and type of distinct organizations mentioned in the SEA research, 
data, and practitioner knowledge networks that were located within, as opposed to outside, each 
state as a proxy measure for the "richness" of the external environment. 9 As we anticipated, 
respondents in State C named the smallest number (13) of distinct organizations in their own 
state environment, a figure that represents only 26% of all the distinct externals organizations 
that were named on the survey (Table 5.4). Within the state, SEA staff were most likely to name 
an IHE, connections made through these institutions' special education assistance centers 
or, in two cases, an IHE's involvement in the SEA's school improvement assistance program. 
Rather, SEA staff relied heavily on organizations located out of state, most of which were federal 
government agencies or centers and national PMAs. Unlike States A and B, which had many 
internal sources for practitioner advice, SEA staff in State C more often turned to national or 
federal organizations for practitioner advice, as well as research. 


9 We did not include journals, or organizations that we could not identify by name, in this analysis. 
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Table 5.4: Location of Distinct External Organizations, by Level of 
Government, Type, and State (Number and percentage of mentions) 



State A 




State B 




State C 





Within State 

National/ 

Federal 

WithinState 

National/ 

Federal 

Within State 

National/ 

Federal 


# 

% 

# 

% 

# 

% 

# 

% 

# 

% 

# 

% 

Government 

11 

48% 

12 

52% 

11 

42% 

15 

58% 

3 

15% 

17 

85% 

PMA 

5 

56% 

4 

44% 

16 

57% 

12 

43% 

4 

22% 

14 

78% 

IHE 

4 

100% 

0 

0% 

2 

67% 

1 

33% 

6 

60% 

4 

40% 

Provider 

1 

25% 

3 

75% 

0 

0% 

3 

100% 

0 

0% 

0 

0% 

Research 

1 

33% 

2 

67% 

1 

20% 

5 

80% 

0 

0% 

0 

0% 

Advocacy/ 

Foundation 

2 

22% 

7 

78% 

0 

0% 

3 

100% 

0 

0% 

2 

100% 

Total 

24 

46% 

28 

54% 

30 

43% 

39 

57% 

13 

26% 

37 

74% 


Contrary to our expectations, State B had the largest number of distinct, in-state organizations 
(30), which represented 43% of all their identifiable external sources. The larger number of 
in-state organizations in State B was due to their many state-level PMAs. Survey respondents 
mentioned 16 such organizations, compared to four to five in the other two states. These 
PMAs represented the typical education groups, such as associations of school boards, district 
superintendents or principals, but also a set of groups representing very specific job functions, 
such as mathematics consultants, state and federal program specialists, and education media. 
State Balso had afewAEAs that they relied upon for advice. 

We see additional variations across the states when the data on in-state organizations are 
broken down by knowledge network. (See Appendix Table 5.5.) All three states drew on their 
IHEs for research advice. SEA staff in states B and C turned to a subset of their PMAs as well, 
while staff in State A sought research information from two state commissions, both of which 
included researchers as members. State A included districts and district networks in their 
practitioner advice networks (seven distinct mentions), and in interviews frequently discussed 
their importance in the school improvement work. In contrast, district administrators were rarely 
mentioned in State B and were not named at all in State C. 
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Policy context and partnership history 

The states' accountability policies, school improvement structures and partnership histories 
also help explain some of the variation that emerged in who SEA staff turned to in their 
advice networks, particularly in the practitioner network. For example, State A has focused its 
accountability and technical assistance on school districts since the mid-1990s, in contrast to 
the school-centric nature of many other state accountability programs and ESEA itself. The SEA 
provides differentiated assistance to districts with low-performing schools through customized 
support for the state's 10 largest urban districts, a regional system of support for other districts, 
and support for district- and content-based networks. An important partner in this work is the 
state's Urban Superintendent's Network, created in the early 2000s in response to a desire by 
urban superintendents to network among themselves and an interest of State A's SEA to engage 
with urban districts. This network has become a place for SEA staff to identify common problems, 
collaboratively address challenges, get practitioner input on proposed policies, and help the 
agency develop guidance and tools used to drive school improvement. One SEA staff member 
noted that this network: 

was instrumental in beginning to give the department a window into districts. Because 
as they began to trust some of the superintendents and learn from the conversations. . . 
they understood that they needed to behave differently... it became the focal point 
within the department.. .to change the way the department worked. 

One outgrowth of the Urban Superintendent's Network was the development of a district 
mathematics network, supported by the SEA's Curriculum and Instruction Department. The 
network became a model for the creation of subseguent networks in English language arts and 
English Language Learner leaders, and for networks supported by the SEA's Regional Assistance 
Teams. 

Staff in States B and C were more likely to characterize districts as a problem to be solved, rather 
than as a partner in the school improvement work. In contrast to State A, States Band C had long 
targeted their accountability programs and their improvement efforts at the school level. But, 
by the time of our study, both states had decided to change their approach and were seeking 
out new ways to enlist district central offices in state school improvement efforts. This decision 
was influenced by NCLB, as well as by research and resource issues. As the impact of NCLB 
accountability ratcheted up, and as State C raised its state accreditation standards, more and 
more schools became identified as low-performing, straining the capacity of the SEAs and their 
service delivery providers. In addition, research syntheses from the Cll, as well as survey and 
evaluation findings in both states, suggested that districts needed to be more engaged in the 
state school improvement work so thattheir policies and programs would amplify, complement, 
or at the least not obstruct, other initiatives. For example, in their review of the work of one IHE 
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provider, the Cll and the regional CAC told State C that having districts "become more involved 
in school improvement is what makes the difference." At the time of our visit, both SEAs had 
recently required district staff members to be part of their state team to monitor and/or provide 
support to the schools. These SEAs were looking for ways to engage districts and persuade them 
to be involved. 

While State A relied more heavily on district staff for practitioner advice than the other two states, 
State B turned to its AEAs. In all, AEAs represented about half of the distinct in-state government 
organizations mentioned by SEA staff. These agencies have a long history of partnering with 
the SEA to provide assistance in such areas as special education, curriculum development, and 
the delivery of professional development. Although their funding had been cut in recent years, 
a subset remained instrumental in helping the SEA construct, modify and deliver different 
components of the state SSOS . 10 School improvement facilitators from these AEAs, for exam pie, 
developed a process for modeling a data-driven cycle of improvement in schools, and provided 
leadership coaches for principals in low-performing schools. In addition, in-state PMAs were 
a major source of practitioner advice in State B. For example, the field services director in the 
SEA attended monthly meetings of an umbrella organizations of unions and PMAs representing 
district superintendents, principals and the school boards to "hear, like, what the buzz is with 
them, what they're supporting, what they're fighting against." 


Qualities of external sources 

In addition to the policy and partnership histories, SEA decisions about whether and where 
to seek out information from external sources was also influenced by perceptions about the 
quality and qualities of these external sources, and the added value that they could bring to the 
work. Searches for the best external sources of information could be comprehensive, and State 
A appeared very active in this regard. For example, SEA staff reported doing an extensive search 
for providers with evidence of success in intervening with low-performing school districts. One 
school improvement staff person recalled, 

We said, 'Who's done this? Is there anybody..?' And there was literally nobody in this 
country. And (our Research Office staff) said, 'There is this entity [in England] that's 
done it at any scale, and clearly has had success. No matter what evaluative angle you 
take, it's a positive report.' So we called them up. 

But more often the SEAs turned to familiar external organizations or individuals because they 
were trusted to be on top of the best research, and could provide quick and ready access to 


10 As we anticipated, while State B has a large number of AEAs, only 9% of these agencies were named in 
the SEA's advice network. 
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the information. Searches thus often extended to external sources with whom staff had a prior 
relationship, or where the quality of the research or other knowledge was inferred from the 
perceived quality and expertise of the source. For example, one SEA staff member in State C 
observed that the CACs were the best resource they had to identify research: 

I think you look at all the research and you have to make a decision. But at least [the 

CACs] bring it to you. I mean, I don't have to go out searching. If I have a question and 

I call... ourstate liaison and say, 1 need some help with this,' I'm going to get that help. 

A manager in State A's Research Department explained why they use the evaluation center 
of a state university to conduct many of their formative evaluations: "We have a history of a 
relationship with the organization. I have to say [they] have done very, very good work for us." 
Another SEA staff person explained that: "we hosted a daylong workshop for districts.. .by Mass 
Insight because of their expertise in extended learning time and increasing instructional time 
within a school day." 

External organizations were also attractive in general because they had the flexibility and 
the resources to address problems or conduct work that the SEAs could not. In discussing 
an independent coalition of organizations that partnered to address educator quality and 
effectiveness initiatives, a State A staff member found it helpful to have a set of stakeholders 
"who believe in a. ..common... and coherent approach, who can work outside of the rule-making 
and regulatory agency experience to pursue these kind of initiatives as thought partners, as 
advocates, and conducting their own research and technical assistance as well." A staff member 
in State C noted that state pay scales limited their ability to directly hire school improvement 
expertise. The PMA foundation which supplied many of their coaches had the resources and the 
flexibility to hire former principals or district administrators to deliver training and assistance, 
and to work with the state to design and modify its approach to school improvement. State 
B turned to external sources for research or services because expenditure rules made it "hard 
for us to spend our own money" to do the research or provide services. Some of the external 
relationships that comprised SEA knowledge advice networks, then, arose in part out of the 
operational necessities of resource-thin organizations operating under bureaucratic constraints. 

External organizations could also provide expertise or research-based tools that the SEAs simply 
lacked. For instance, both States B and C had very limited capacity for high school improvement 
and guidance, due in partto staff cuts. State C had eliminated the position of high school director 
in their Curriculum and Instruction Office, leaving the unit to an individual with an elementary 
school background. As more high schools were identified under state accountability for low 
graduation rates, State C reached outtothe National High School Center and a national provider 
organization to help them develop tools to assess whether middle and high school students 
were at risk of dropping out. 11 Similarly, State B struggled for many years without any expertise 


11 In contrast, the SEA in State A developed an early warning indicator system based on its own research 
and with its own staff. 
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in that area as well, despite a gubernatorial initiative focused on high schools. The SEA similarly 
turned to the National High School Center and its regional CAC to provide guidance in their 
work. State C relied heavily on its commercial vendors for psychometric expertise because it did 
not have in-house staff with those abilities. 

The expertise carried by external organizations or individuals also provided an outside 
perspective on whether state efforts were within the bounds of best practice-an important 
metric for state agencies and policymakers engaged in often uncertain work-and to help them 
translate research, or laws and regulations into a course of action. For this reason, SEA staff often 
sought to learn about other states' initiatives, and to interact with SEA professionals engaged in 
similar work. At the national or federal level, the U.S. Department of Education, ASCD, and the 
CCSSO were common sources for such practice knowledge. A respondent in State C noted that 
U.S Department of Education officials provided information on: 

how other states might be implementing something that we're struggling with. 
They're the greatest source of information, you know, rather than us calling around to- 
other states, they've talked to the other... 50 whatever states, and can give us sort of a 
snapshot of what's happening across the country, and who might be doing something 
in an innovative way. 

Similarly, when designing a new system of regional support for its low-performing school 
districts, SEA staff in State A turned to its regional CAC that had been conducting some research 
about how assistance was being provided regionally in other states. 

In addition to material or web-based publications and conferences, these external organizations 
often involved SEAs in formally organized networks. CCSSO has undertaken many initiatives 
to engage state staff in project or on-going, topic-oriented groups, including one for school 
improvement facilitators. One staff member in State B said that he participated in six such 
collaborative groups sponsored by CCSSO, and was motivated to join so many because they 
enabled him to build relationships with colleagues doing similar work and wrestling with 
similar design and implementation challenges. Such networks with job-alike professionals 
helped them exchange ideas and information about how to move forward on ambiguous or 
vaguely specified research and policy mandates. For example, when changes in federal policies 
were introduced, like the 2010 guidance on Title I School Improvement Grants, one SEA staff 
member reflected: 

I don't think any of us had any idea what we were doing. And [CII's Pacesetters] allowed 
us monthly to meet with all of the nine states... And with Sam's help, we made our way 
through the School Improvement Grants requirements in order to understand them.... I 
have no way of getting that kind of capacity to my staff, building my knowledge let 
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alone staff knowledge. So it allows us to build a...SSOS team that understands these 
requirements well and can interpret them and make sense of them to get it into practice. 

Some SEA staff noted that interactions with other state professionals were an important way for 
them to validate or disconfirm research findings. For instance, one individual from State C said: 

because they are doing it with schools, and state agencies that we are familiar with,... 
we... .get to talk to the people who are actually involved in it. So I think that increases, 
in your mind, the validity because you are hearing it, as well as the actual reports.. ,[l]t's 
a trust thing. ..there is a lot of stuff out there. And I think we have to be careful about 
we're not getting stuff that is not valid and that's not research-based. 

Similarly, a staff member in State A's SEA noted that when he saw a demonstration of a 
sophisticated diagnostic assessment and assistance program that had received significant 
funding from the National Science Foundation: "I said, I better pay attention to this and see 
what he's doing. Kind of interesting. I said [to the developer], you might want to shop it around. 
I'd like to hear what other people in other states say about it." 

SEAs also gravitated to these external sources to lend credibility to the work. External 
organizations, they believed, could be more neutral purveyors of research or evaluation feedback 
than the SEA itself, an asset if targeted schools or districts are wary of the state reform agenda. 
The PMAfoundation in State C, for instance, was symbolically valuable because, as one SEA staff 
member explained: "you don't come with the department's agenda, you come with an open 
agenda." One former superintendent, now an IHE faculty member, was recruited because he 
was a well-respected by district administrators, and the SEA felt that his input would build district 
confidence in the efficacy of their school improvement endeavor. 

Finally, external intermediaries frequently played an important role in synthesizing and 
packaging research to make it useable and useful to SEA work. As an administrator in State A 
explained: 

Because our work [in my office] is so huge, I've relied on consultants and organizations 
that can capture and summarize... research so that we can figure out, focus on how 
we're going to use it to inform our work. Sometimes I get that help in the form of a piece 
of research or book, but that's rarer. It's usually a conversation and sometimes it's a... 
model or an example or a consultant that has had experience with this work. 

A respondent from State B echoed: "We get [research] all the time from external sources, but do 
we have the time to sit and figure it out and apply it becomes the issue?....That's where Cll comes 
in handy because they will do that kind of work and then we get to choose to accept it or reject 
it." We explore the role of this synthesized knowledge, or "research designed for use," in SEAs 
school improvement work further in Chapter 6. 
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Conclusion 

Our study makes explicit the intermediary organizations that populate the school improvement 
advice networks of the three study SEAs. The survey data provides a snapshot of their 
environment in one year, a picture that undoubtedly changes as the policies, problems, 
structures and resources inside and outside these agencies shift. But it is an important step in 
gaining an understanding of who, how many, and what types of organizations SEA staff rely on for 
knowledge resources, since external groups play a major role in providing evidence, especially 
research-based evidence, that flows into the school improvement initiatives undertaken by SEAs, 
a portfolio of work that has grown substantially over the last two decades. 

We learned that a small proportion of SEA staff engaged in school improvement work turned 
to a large and diverse set of external organizations, especially in the government sector and 
across professional membership associations. These organizations became important sources 
primarily for research and practitioner advice; data-related advice is more exclusively an internal 
pursuit. 

Federal agencies or federally funded research and technical assistance centers, as well as 
national PMAs, were major sources for research information. Practitioner knowledge was more 
often acquired through in-state institutions, particularly from districts in States A and regional 
agencies and PMAs inside State B. State C had a smaller set of in-state sources of advice, and 
so national PMAs played a more significant role in providing the kind of experience-based 
knowledge that practitioners represent. Similarly, while the Cll and other federal CACs were key 
sources of research knowledge in States B and C, they were not significant in State A. 

These and other differences in the sources that came to populate or dominate in the SEAs 
intermediary environments were due to a combination of factors. The internal capacity of the 
SEAs, such as the robustness of their Research Office, the numbers and types of staff engaged in 
school improvement, and even the bureaucratic rules and procedures about hiring influenced 
where they turned and, to some degree, why they turned to particular external sources for help. 
External organizations could help fill gaps in SEA knowledge and skills, and provide staff in 
areas where they simply lacked sufficient people or time for work. 

The stage and focus of the problems that they were addressing in their school improvement 
work, the structure of intermediaries inside states, and state and federal accountability policies 
also played a partin which external organizations became important sources for SEA staff. State 
A was at a relatively stable phase of policy development, and had a long-standing focus on 
districts as the primary unit of accountability and support. They were more likely to turn to 
external organizations to pull in research and tools on an as-needed basis, and to rely on their 
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district practitioner networks to provide feedback on policy design. On the other hand, States B 
and C were grappling with problems of practice that led to some significant reconsideration of 
theirdesigns. Forthem, having external partners who could inject research and other knowledge 
resources as problems were discussed was important. This more fluid and open-ended process 
is what the Cll network, the Academy of Pacesetting States, provided. 

Finally, these SEAs gravitated to external sources that were familiar and trusted, generalizing 
expertise in one area to expertise in others. They also turned to external sources that they 
believed provided credibility to their work, validated that their approach was in line with other 
states' policy or practice, and provided easy access to research, especially in the form of research 
syntheses or research-based tools designed for use. 
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CHAPTER 6 

The Use of Knowledge in 
School Improvement Policy 

Lead Authors: Margaret E. Goertz and Diane Massell 


Studies of research use in other contexts have found that policymakers draw on multiple forms 
of evidence in their decision-making processes (see for example, Coburn, Honig & Stein, 2009, 
and Honig & Coburn, 2008, for a review of this research). As we discussed in the Introduction, 
research also suggests that the use of multiple forms of knowledge is critical in contextualizing 
and elaborating on research for action. 

As described in previous chapters, SEA staff in our study used a range of research, data and 
practitioners' knowledge to inform the design and implementation of their school improvement 
frameworks (SIFs), processes, tools and other forms of school improvement assistance. This 
chapter delves more deeply into SEA knowledge use to address our third research question: 
What role does research-based and other forms of knowledge play in states' strategies to 
improve districts and schools? Following on Coburn, Honig, and Stein (2009), we characterize 
"use" as the types and forms of knowledge that SEA staff report considering in their decision- 
making and that are embedded in the states' school improvement policies, processes and tools 
(the "what" of knowledge use). 

Our findings about the use of knowledge in school improvement policy in our three study 
sites are shaped by our data collection methods. We did not observe the actual decision- 
making processes. Rather, like some other studies of evidence use, we based our analyses on 
retrospective accounts by the SEA staff of how specific decisions were made and with what kinds 
of evidence. Although we did not observe discussions of specific pieces of evidence as SEA 
staff discussed and made decisions about school improvement policy and/or designed school 
improvement tools and processes, we did triangulate respondents' reports with examination 
of documents in each state to see whether, to what extent, and what kind of research and other 
types of evidence is cited in school improvement policies and tools. 
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We begin the chapter with a description of the types and forms of knowledge that framed our 
analyses, followed by an examination of knowledge use in the design and implementation 
of school improvement policies in the three SEAs. Finally, we explore why SEA staff turned to 
particular knowledge, factors that include certain qualities of the evidence, but that extend 
beyond its inherent characteristics. 

Types and Forms of Knowledge 

Many studies of knowledge use have not distinguished between different types of evidence, a 
blurring that may contribute to a lack of clarity about the nature and scope of knowledge that 
actors bring to bear on particular problems. To address this issue, we identified three types of 
knowledge-research, practitioner, and other evidence-to develop survey questions, code open- 
ended interviews, describe networks, and categorize materials used in school improvement 
policies. As described in Chapter 1, we defined research-based knowledge (RBK) as research 
findings that have been to varying degrees "collated, summarized and synthesized," then 
presented in ways that provide "empirical or theoretical insights or make them otherwise 
informative" (Davies & Nutley, 2008). We included in this category published original research, 
research syntheses, summaries or meta-analyses, and formal evaluations of internal or external 
programs.These may be presented in journal articles, books, materials, or other media developed 
by intermediaries. We distinguished these more prevalent forms of RBK from forms that are 
designed for use in practice; that is, models, programs, protocols or other tools that embed 
research-based practices in guides to action. We defined other evidence-based knowledge 
(EBK) as data, facts, and other information relevant to the problem of school improvement, 
such as state test scores, the distribution of qualified teachers, and formative feedback loops on 
implementation, and practitioner knowledge, as the information, beliefs, and understanding of 
context that practitioners acquire through experience. 

Knowledge Use 

To get an overall picture of evidence use in the three SEAs, we asked survey respondents how 
often they used different types and forms of knowledge to inform their work. SEA staff reported 
that they used a range of information-research, data, advice from colleagues, and advice from 
practitioners outside the SEA-to inform their work, although the type of evidence they used 
most often varied (Table 6.1). A majority (54% to 65%) of SEA staff reported using data on 
schools in their state on a regular basis, that is, either daily or once or twice per week. Reported 
use of data is not surprising given the keen focus of federal and state policy on performance- 
based accountability, and subsequent high stakes interventions in schools and districts based 
on those numbers. In States A and C, smaller percentages reported using research (38% to 
39%), research-based guidance (34% to 44%), or advice from practitioners outside the SEA 
(32% to 39%). Respondents in State B reported more frequent use of these sources. Within 
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each state, SEA staff used the results of program evaluations the least often. Only about 10% 
of respondents, on average, reported that they "Never" used the different types of information. 

Analysis of interviews and documents shed light on the specific information SEA staff used 
in their school improvement decision-making and related activities. Examples of the use of 
research, research desig ned for use, other evidence, and practitioner advice are presented below. 

Research 

SEAs used research to inform their school improvement policies, but tended to access research 
that was packaged and presented in short summaries, syntheses, or meta-analyses. These 
traditional forms of research were included most extensively in the development and revision of 
the SIFs that were central to their overarching strategy. The district standards and accountability 
system in State A date from the late 1 990s. In 2008, the SEA u ndertook a review of the standards 
that are the foundation of its district and school accountability policies, school improvement 
planning, and technical assistance in an effort to make the standards more manageable, 
coherent and useful, and to update the underlying research. SEA staff pulled from a variety of 
sources for this review, including extant research, stakeholder groups, and working groups tasked 
with developing a new teacher evaluation system (for research-based human resources and 
professional development indicators) and a framework for supporting children with behavioral 
health needs (for research-based student support indicators). The SEA and the State Board of 
Education's advisory council also asked their Regional Education Lab (REL)to review the research 
on school effectiveness "in order to establish an independent body of research to verify these 
[school] indicators" (REL document). The resulting 92-page literature review included research 
summaries, research syntheses and meta-analyses that were usually produced by researchers, 
and practice guides published by the U. S. Department of Education's Institute of Education 
Sciences. There were fewer citations of specific, individual research studies. The review provided 
a page for each reference, composed of a brief summary of the document, its methods, findings, 
and recommendations. 

In State B, the impetus to create a new school-level improvement framework in 2006 was driven 
by the perceived inability of districts to produce meaningful plans to address Title I schools not 
making progress. According to staff in the SEA's School Improvement Department: 

We were getting nothing. I mean, it was just awful stuff. And then we would go back and 
we'd say, 'Let's see what you've done?' And they would have done nothing. They may or 
may not have spent the money. 

SEA staff engaged school improvementspecialistsand othereducatorstodevelopthe framework 
and companion planning documents, and drew upon the literature reviews and information 
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Table 6.1. Types and Forms of Knowledge Used in Work 1 


"How often do you use the following to inform your work?" Percent of respondents answering 
"Once orTwice a Week" or "Daily or Almost Daily" by Knowledge Type and State 



State A 

State B 

State C 

Research (e.g., published original 
research, research syntheses or 
summaries, meta-analyses) 

37.5% 

51.6% 

38.8% 

Program evaluation (e.g., published 
or unpublished evaluations of 
programs or policies) 

19.8% 

38.1% 

24.1% 

Research-based guidance (e.g. books, 
best practice guides, protocols or 
other tools written for practitioners) 

34.0% 

54.5% 

44.3% 

Data on schools in your state (e.g., 
student outcome data, administrative 
data such as the distribution of 
highly qualified teachers, etc.) 

65.0% 

61.6% 

54.3% 

Advice from colleagues within the 
SEA 

83.5% 

82.4% 

75.6% 

Advice from practitioners outside of 
the SEA who have knowledge of how 
your programs work within your state 
or in other states 

39.2% 

45.2% 

32.5% 


1 The full range of frequencies is reported in Appendix Table 6.1. 
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acquired from two other state initiatives on effective schools and student achievement. The 
framework includes 92 different citations to documents that are a mix of research studies, 
research syntheses, and many practice guides. By the time of our visit in 2011, State B's 
framework was widely viewed as the touchstone to which all other strategies and tools should 
relate; staff perceived the framework to be held in high regard, and, therefore, not itself a 
document to be revised even when other aspects of the system were undergoing review. Staff 
attributed its status to the quality of the research, although the research had not been revisited 
since the framework was originally published. 

Both States A and Balso used the research cited in theSIFs to validate their school improvement 
activities. While the literature update in State A did not lead to any substantial revisions in 
their approach to the work, it was used to legitimize solutions and develop shared language 
and understanding (Robinson, 1988, and Manheimer, 1995, as cited in Coburn, Honig, & Stein, 
2009). State A, for example, posted the REL's review of school effectiveness literature on its 
website. As one SEA staff member explained: 

We've used research [for the district and school standards] explicitly because we want 
to be transparent about it so that it can guide, so that people understand they're being 
held accountable to things that research tells us are important. But also so that... we 
can tie the assistance around some of what research is telling us as well. 

Another SEA respondent noted that State A also embeds research in their school improvement 
tools: "so that anybody who downloads the tools from the website can see the connections to, 
well, okay, this is Marzano's work, so this is why we want to do this." 

Similarly, the research base underlying the SIF in State B was seen as a way of conceptually 
integrating the SEAs activities and developing understanding across organizations involved in 
the work. An SEA official in State B described the role of their framework and its research base 
in developing an improvement process that schools would engage in with an external provider 
team in this way: 

We made sure we took each of the five strands [of the school improvement framework]. 
And if you think of what this big picture is, how can we use the research from these five 
strands to do that? So that was the whole basis of [our new component of support].... 
But I think the fact that we based it so solidly on research in the first place, I think that 
really helped as we went along that it didn't ever need anything major to change. 

The three states also referred to traditional research in other components of their improvement 
supports, although in general these were not as extensive as the frameworks. State A's guidance 
document for creating professional learning communities (PLCs) includes references to research 
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and to research-based curriculum units by the National Institute for School Leadership (NISL), 
a co-developer of the guidance, for each stage of the process. Similarly, the SEA highlighted 
research on data-driven inquiry and action cycles in their guide for district data use. In some 
cases, however, connections to the research literature were more indirect, with documents citing 
other documents that in turn referenced research, or something claiming to be research-based. 


Evaluations 

States A and B used the results of formal formative and summative evaluations of their school 
improvement programs, as well, to inform their work. These states gave priority to and pursued 
grants for evaluations of their school improvement activities, although in some cases they were 
required to conduct evaluations as part of a larger program award. By comparison, State C was 
less active in seeking out formal evaluations of its own work in this domain, in part because of 
resource constraints, although they did use evaluations of other programs or interventions to 
select vendors (see below). State C also perceived the School Improvement Office's work to 
be successful, which certainly contributed to the high autonomy this office was afforded and, 
perhaps, to the lack of a perceived need for formal evaluations. Their evidence of success lay in 
the fact that many of the low-performing schools had made enough gains to exit accountability 
designations. The SEA did make frequent adjustments to their school improvement approach 
in response to feedback from providers or educators in the field, or when changes in federal or 
state policy compelled them. 

State B contracted with a national research organization and a state university to evaluate the 
implementation and effects of its statewide system of support (SSOS).The first evaluation focused 
on the fidelity of implementation, interaction and coherence of the SSOSs, and its impact on 
student achievement. The Center on Innovation and Improvement (CM) helped to design the 
evaluation request and vet the quality of the designs proposed by external applicants. Among 
other things, the evaluation confirmed the SEA's perception that it needed to secure districts' 
engagement in the work, and prompted the state's modifications of rules to do so. In addition, 
SEA respondents reported that the evaluation stimulated conversations about where they could 
make the system more integrated and cohesive. As a result, the state decided to make the area 
education agency representative more of a case manager who meets quarterly with all the 
other service providers to make sure they are working together. The SEA later contracted with a 
different national research organization to conduct a formative and summative evaluation of the 
state's Title I School Improvement Grant(SIG) program. 

At the time of our data collection, State A had designed and was starting evaluations of 
implementation of its multiple Race to the Top-funded projects, Title I SIG and Teacher Incentive 
Fund grants, and related initiatives. To streamline the evaluation process and reduce data 
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collection burdens on participating schools and districts, the Research Department grouped the 
projects into nine topical evaluations (e.g., teacher evaluation, low-performing schools), most, 
if not all, of which would be contracted to outside evaluators. The evaluations were designed 
to provide formative feedback to inform the ongoing implementation of the initiatives, as 
well as summative evaluations at the projects' end. In previous years, the SEA had evaluated 
state-funded extended learning time, school reform, and school leadership training initiatives, 
reporting results to the State Board of Education and legislature. 

State A was also in the process of evaluating the implementation of its regional system of 
support to low-performing districts. An in-state university-based evaluation center surveyed and 
interviewed district and school stakeholders and SEA-affiliated staff who provided assistance to 
eligible local entities annually. The reports to date (through the second year of implementation) 
focused on district and school leaders' reported use of, satisfaction with, and perceived impact of 
the assistance on changes in district culture, capacity, and practice. The report identified the need 
to improve overall communication between the districts and the SEA and to offer differentiated 
resources to districts based on their needs. The evaluation also highlighted challenges districts 
faced in responding to the growing number of education reform initiatives in the state. The 
findings were shared with and discussed by the managers and staff of the Regional Assistance 
Teams and senior staff in other parts of the agency. 

The three SEAs used external evaluations of instructional programs or other interventions, such as 
those included in the What Works Clearinghouse, in their decisions. States A and C also reguired 
technical assistance organizations to provide evidence of effectiveness to qualify as approved 
providers. State C, for example, provides a list of pre-approved lead turn-around partners for its 
lowest-performing schools, enabling local sites to by-pass a lengthy procurement process and 
thus seeking to encourage selection of a partner that meets state criteria. Specifically, they 
created an RFP using Mass Insight Education and Research Institute's The Turnaround Challenge 
(Calkins, Guenther, Belfiore, & Lash, 2007) and supporting resources, and vetted models based 
on a rubric that included evidence of program effectiveness. Seventeen programs applied, and 
four were chosen. State A compiles a list of recommended providers that districts can access for 
assistance in turning around low-performing schools. Partnering with these approved providers 
is voluntary. The SEA's Research Department solicits applications from external providers under 
each of the state's school standards (e.g., "maximizing learning time," "effective use of data," 
"social, emotional and health needs"). Applicants mustsubmitinformation on and are reviewed 
against six criteria, including the theory of action underlying their services, plans to achieve 
measureable outcomes and demonstrated record of effectiveness in underperforming school 
environments. Only a small number of applicants make the final cut. 
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Research designed for use 

While foundational documents, such as the SIFs, contained some references to original 
research, the states in our study more often relied on research tools and strategies that were 
at least purported to be research-based. These tools and strategies provide very specific and 
concrete guides to action, and are what we call "research designed for use," that is, research that 
is actionable, feasible and addresses a pressing problem of policy or practice. 

For instance, like State B, State C recognized that the school improvement plans (SIPs) submitted 
by low-performing sites were often premised on very localized notions of good practice rather 
than solid research on effective change. State C also observed that the plans cobbled together 
practices without a coherent theory of action, and that many sites presented the same plans year 
after year, suggesting that schools were treating the required plans as a bureaucratic compliance 
check rather a template for meaningful change. As we have described in other chapters of 
this report, State C found that the CII's Handbook on Restructuring and Substantial School 
Improvement (Walberg, 2007) offered a practical, research-based solution to these problems of 
practice, and used that resource extensively in their work. 

Again, the central mission and purpose of the federal comprehensive assistance center (CAC) 
system was to provide research to SEAs to help them implement NCLB; it was the most recent of 
a long line of federal initiatives to disseminate research to the field. Many of the prior initiatives 
were not seen as successful because educators did not read or use what was disseminated; 
the research was criticized as esoteric, irrelevant, ambiguous, and lacking in guidance (see, for 
example, Flood, 2002). Recognizing these issues, at the outset the CII's director said that their 
job was not just to provide research, but to help SEAs "know how to integrate [research] in their 
work and solve their problems." The Handbook is a strong illustration of research designed for 
use, and the process they used to create it reflects this purpose. Specifically, the Cll identified 
leading scholars in the field and asked them to synthesize extant research on specific topics, 
such as districts' role in improvement, or strategies for changing and monitoring instruction. 1 
According to a Cll staff member: 

We got people we thought would be strong. And then we gave them guidelines. We 
said, we want 20 pages where you synthesize the research and you boil it down to 10 
action principles. So, we really wanted the researchers to not just give us a big research 
chapter, but to synthesize that research and boil it down to I think it was ten action 
principles for a superintendent, for a school board, for a principal, for a teacher team, 
for a teacher in a classroom. ...Then we took the action principles and developed the 
indicators from them. ...For all of these we would then take it a step further and try to 
develop some kind of technical assistance process from them. 

1 The Handbook points out that the extant research on these topics did not include any "gold standard" 
randomized controlled studies. 
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The Handbook concludes with a consolidated checklist of indicators on action steps that 
educators at different levels of the system can use to assess program strengths and progress, 
and were subsequently taken up by many other SEAs. 

The importance of this "guidance for practice" component to our SEA staff cannot be 
underestimated. Referring to this and other Cll material, a staff member from State B said, 
"...if you are a high-priority school, what do you need to be looking at? [Cll] just published a 
whole series of very practical monographs that you can go to and answer this question. ...And 
it's practical as opposed to esoteric." Another State B staff member noted that John Hattie's book, 
Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement (2008), had 
"taken off like wildfire" with their school coaches, in part because it presents a research-based 
dashboard for comparing innovations, a tool these providers could use to help their schools 
evaluate needs and select effective practices. SEA staff in both States A and B used the research- 
based Instructional Rounds in Education (City, Elmore, Fiarman, &Teitel, 2009) to focus their own 
and their districts' school improvement strategies more specifically on classroom observations 
of instruction, student learning and academic content. School improvement staff in State A 
reported that some districts they work with were guiding their own work with books like Jim 
Collins' Good to the Great (2001), or Results Now (Schmoker, 2006), published by ASCD. 

Research designed for use was clearly valued and preferred by SEA staff in all of the states. But 
it is important to recognize that even this form of research was typically further adapted and 
modified to the specific policy or improvement frameworks in place, and the needs of context 
as interpreted and understood by SEA staff. So, for example, although the Handbook had 
already synthesized research and produced very concrete guidance, State C decided to modify 
the materials in ways that they perceived would be more relevant and useful to their schools. 
Specifically, they thought that the large number of indicators of successful practice would be 
too overwhelming to struggling schools. (The Handbook contains 12 checklists with more than 
100 indicators.) So, with the knowledge and assistance of their own practitioners and delivery 
system coaches, the Cll, and their regional CAC, the SEA created a streamlined set of indicators for 
their schools. And, because of their increasingly limited budgets, travel restrictions, and staffing 
constraints, they encouraged the Cll to create a web-based platform to more inexpensively 
monitor progress while maintaining frequent contact with local educators. 

Building on this positive experience, the Cll and other CACs continued to be a major provider 
of research designed for use in State C, often following a similar adaptation process. For 
instance, as the state mathematics standards were raised and more schools were identified 
for school improvement, the SEA wanted to expand on their efforts to engage districts in the 
improvement work. The SEA also decided to tailor the kind of technical assistance that districts 
and schools received based on a needs analyses. Their regional CAC presented a research- 
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based model of change to address this challenge. SEA staff met over a three-month period 
with their service delivery providers and this regional CAC to read and discuss and build on 
this model. One member noted that she and other SEA staff read the underlying research 
cited in the model, such as Designing Social Systems in a Changing l/l/oc/cf (Banathy, 1 996) and 
Readiness for Change [ Fixsen, Blase, Horner, & Sugai, 2009). The former is a book that presents 
a comprehensive review of different approaches to systems change, and provides theoretical, 
empirical and reflective guidance on how to practice it. The second is a brief, based on a synthesis 
of implementation research, that offers structured advice to state and district professionals for 
assessing and assembling readiness for scaling up innovations. The SEA focused in on one 
particular component of the CAC change model and created a version of it for its own districts. 

SEA staff in State Asimilarly adapted research designed for use to meet the needs of their districts. 
Working with the NISL and a group of urban districts, the SEA developed a PLC document that 
provides guidance fordeveloping and strengthening instructional teams atthe school level. The 
guide includes references to research-based curriculum units from NISL and related research, 
and SEA resources, for each stage of the process. The SEA drew on materials from the Council 
of Chief State School Officers' Decision Support Architecture Consortium (2008) to identify and 
disseminate promising district practices. 

Other evidence-based knowledge 

All three SEAs strived to become more data-driven organizations with regard to both their 
internal work and their work with schools and districts. Each state houses an education data 
warehouse containing basic student and school input and performance data that are submitted 
by and available to school districts. The three states have developed data analysis tools to 
help schools and districts use these data in school improvement planning and to support SEA 
monitoring and assistance. State A, for example, requires all low-performing districts to develop 
school and district improvement plans that are reviewed and monitored on a regular basis by 
SEA staff. With too few staff to conduct the legislatively mandated number of annual audits, the 
SEA created a web-based data analysis tool that can be used for both planning and monitoring 
purposes. As a SEA manager explained: 

What we needed to do was to provide districts with some of the quantitative data and 
quantitative analyses that district reviews would normally provide. Why can't we do 
that annually and why can't we do that for every district?.... Then we realized we could 
make it interactive... .And then we realized it could straddle the line. It could be on the 
one hand an accountability tool that we use,.. ..but we could produce it in a way that 
could be a self-assessment tool. ...It's an unbelievably popular tool, and getting more 
popular. 
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Over time, the SEA expanded this tool to include data on finance and staffing. As another SEA 
respondent noted: '[Our deputy commissioner] felt if we collected it, districts should be able 
to have access to it." Data analysts assigned to each of the state's Regional Assistance Teams 
support school and district staff in use of these tools. 

State C streamlined some of its reporting systems to encourage districts to "[stop] doing 
checklists and [start]... looking at school improvement processes." The SEA gave districts a data 
dashboard tool that takes data like student attendance and nine-week grades from districts' 
student information management system and puts it into an analytic framework that SEA staff 
can view as well. As one SEA school improvement staff person explained: 

We can then ask them specific process questions about, 'What does your data tell 
you about your attendance intervention? What does your data tell you about your 
interventions for reading?' Having them really answer more process questions.... Does 
that make sense? We really worked on that. 

Staff in State B's Departmentof Assessment and Accountability oversee professional development 
projects for school coaches and other teacher leaders on the use of formative assessment data 
to improve instruction. A data specialist in State C's Assessment Office helps districts and school 
improvement coaches interpret data on districts, students, and teachers. 

The SEAs have used the data they collect as they monitor school improvement activities to identify 
both problems and best practices. As discussed elsewhere, poor quality and/or rarely used SIPs 
triggered a re-examination and revamping of SIFs and processes in all three states. In State A, 
site visits and monitoring reports identified the need for more targeted technical assistance 
tools and support. For example, school improvement staff working with a set of urban districts 
found that it was difficult for them and for district staff to capture what classroom practice looked 
like. As one SEA facilitator described: 

We had used a variety of approaches to get in and see what was happening at the class 
level and help districts do the same. But indeed variety was the key term there. So we 
decided, why don'twetalkabouta learning walkthrough process that could work? Why 
don't we figure out what professional learning communities might look like, what a 
visitor to a PLC might look for, look at? 

Work with low-performing districts led the same SEA to identify superintendent turnover, 
dysfunctional school boards and lack of communication among district administrators, school 
boards and their unions as deterrents to reform. This led to the development of a superintendent 
induction program and a training program for school board members. 
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State C required targeted sites to upload data, planning and implementation information into 
the Indistar® system. SEA staff reviewed this information to identify needs or problems, and 
to tailor the content of their training and coaching to the needs or problems that appeared 
across multiple sites. Information from these reports could also lead to modifications in state 
strategies. For instance, reports suggested that schools were not using data to revise instruction. 

We're constantly looking at that data that they're turning in to me saying, 'Okay, what 
do I see in this?' ... 'What is it I see across all this data with these 17 schools?' It's the 
same. They're not using it to change instruction in the classroom." 

The SEA consequently pulled together SEA data staff to develop a new format that requires 
schools to specify what they were changing in the classroom as a result of their analyses of 
student performance data. The SEA also extended the Indistar ® training to provide more time 
for reflection on the process and the use of data in the work. 

Finally, the School Improvement Department in State A's SEA has used monitoring reports 
prepared by its Accountability Department to identify promising school improvement practices 
in schools that were required by state law to engage in a formal turnaround process. Monitors 
conduct annual multi-day site visits and prepare reports centered on the state's district and 
school standards. An external evaluator was hired to compare reported instructional and 
organizational practices between schools making significant and schools making little or no 
student achievement gains. The resulting report, which drew on related data sources as well, 
identified a set of "emerging practices," such as having an instruction-oriented principal, that 
were common across higher growth schools, regardless of their turnaround strategy. When 
asked about the report's use, a senior SEA staff person commented: 

It's one of those documents that say, you know, it doesn't tell you anything you probably 
didn't know, but it's really helpful to confirm it. It's not about what you say you are 
doing, it isaboutwhatyou do. ..and what conditions are that you putin place to enable 
those things to happen. ...We use that report across the systems to say, take a look at 
this. Try to imagine what your schools are doing and see whether you can begin to get 
to the point where you're using these kinds [of practices]. 

Practitioner knowledge 

The SEAs also solicited the engagement and feedback of practitioners in the field to define, 
develop or refine school improvement policies and strategies. By practitioners, we mean local 
educators (teachers, administrators) and service delivery providers with knowledge of the field 
as well as educators or system actors in other states. The essential characteristic is that this form 
of knowledge arises largely from the source's experience. 


1 42 CONSORTIUM FOR POLICY RESEARCH IN EDUCATION | Cpre.org 


The Use of Knowledge in School Improvement Policy 


Educators in the field very frequently engaged with SEA staff in State A to develop a wide array of 
school improvement tools and supports, bringing in their own knowledge and local experiences 
to research or research designed for use. The Urban Superintendents Network created in State A 
over a decade ago, and meeting on a monthly basis since then, has been key in helping the SEA 
identify common problems, provide feedback on current initiatives, and, at times, collaborate 
on creating strategies or tools to address new challenges. For example, they used the network 
to identify common needs and develop tool kits, and to pilot and refine state initiatives, such as 
the learning walkthrough protocol and the PLC document described above. Similarly, the SEA's 
school improvement, data, technology, and curriculum and instruction offices collaborated with 
a national consulting firm and five urban school districts to create a district data team toolkit. 
Drawing on a research-based data-driven inquiry and action cycle, the toolkit provides detailed 
modules and rubrics to help districts engage in inquiry and use data to inform district- 1 eve I 
decisions. 

All of the SEAs noted efforts to solicit practitioner knowledge from professional membership 
associations (PMAs), some of which were deeply involved in the delivery of state school 
improvement supports. Coaches and mentors from the area education agencies (AEAs) and 
their membership association were essential partners in the design and delivery of components 
of State B's SSOS. SEA staff in State B also went out to these associations to "take the pulse," 
persuade, or identify problems. "When we go to the organizations, it's more, 'Here's a new 
policy, now tell us all the things that are going to be a problem about it. Give us the feedback, 
let's work it through.'" School improvement facilitators who work closely with targeted schools 
also provided regular feedback on how the initiatives were playing out it schools. 

State C turned to its coaches, largely retired educators who were hired and trained by a 
principals' membership association, to deliver supports. They were also involved in the design 
or modification of training and tools. This knowledge of the field helped the SEA modify and 
adapt its approach to the needs of different districts in the state. A senior school improvement 
staff person noted: 

I looked for people with a lot of district experience... in a variety of places from ... 

around the state, gave me some feedback of, 'That's not going to work in the southwest. 

This might work here, but I don't think you're going to get that to work in northern 

[State C].' So they allowed me to help think about a process that would meet the needs 

of all schools, but be flexible enough to meet the diverse needs of schools as well. 

As discussed earlier, State A drew upon the state's superintendents' association to work with 
university faculty to develop the induction program for new superintendents. Practitioners from 
national PMAs were engaged, particularly in State C, which had relatively few in-state sources for 
this kind of knowledge. 
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Each of the SEAs solicited feedback from local educators receiving supports, although the extent 
to which they reported engaging in these activities varied from state to state, as did the scope 
and formality of the feedback process. The SEAs reported using focus groups, quick surveys, or 
regular meetings to solicit this kind of knowledge. State C, which had no formal evaluation of 
its system of supports like States A and B, relied on annual surveys of its providers and educators 
in targeted schools to gain this information and make adjustments in the work. A focus group of 
principals in State B led to a significant revamping of the program from prescribed supports for 
schools in specific phases of not making AYP, to supports based on identified needs. 

Finally, SEA school improvement staff actively sought advice and information from SEA staff 
outside of their own states. These staff provided alternative perspectives and insights on federal 
policies and regulations, and were a source of ideas about research, design and implementation. 
In other words, they filled in the gaps from research or policy to day-to-day practice. Reflecting 
on a conference call hosted by the U. S. Department of Education with multiple states, one State 
C staff member said: 

What helps there is that there's a question and answer period at the end where we can 
hear what other states are doing, thinking. Their interpretations. And it really helps us 
to think about how we see things and move outside of our sort of, 'state vacuum,' and 
look at nationally, what are some of the questions that are being asked. 

An SEA assessment person from State B who was involved in a consortium of state assessment 
staff put it this way: 

Personally, what I try to continually gather and carry forth and to inform those 
conversations are what's happening in terms of assessment and accountability around 
the country, so from the perspective of other state agencies. So, who's building what 
types of tests, what kind of decisions about how they're going to measure content and 
how they're going to use those scores in various accountability pieces and things like 
that. That's then, if I had to pick one thing that's been the most helpful for me in terms 
of my thinking, it's what research are my peers around the country drawing from that I 
could also utilize, what decisions they're making, how are they handling various policy 
challenges in their states. 

The states also turned to their regional or national CACs and to the federal government to collect 
this kind of information. The Cll's convening of states in the Pacesetter's Academy was a strong 
source of state practitioner knowledge for SEA staff in both State Band C. 

We got to talk about our statewide system of support, what each other was doing, what 
was working, what wasn't, what was problematic and how it goes. It was reassuring 
that we were all doing a lot of the same things. We were all facing a lot of the same 
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problems, so if nothing else, we had some company in our work that we were doing. 
And then also got a lot of great ideas of how we can improve. So that was really- those 
kinds of things kind of lead to a perfect storm of okay, it's time to change it to a needs 
based. How do we do this and who helps us with it? 

Qualities of Evidence 

The research and other types of knowledge that SEA staff searched for and used often had 
particular qualities. Some of these qualities pertain directly to the knowledge and evidence 
itself, while others are related to the characteristics of the search process or of the source of 
information. Here we elaborate on the qualities that emerged most often in our discussions 
with school improvement staff, and that are often interrelated: 1 ) specified and comprehensible; 
2) contextual validity; and, 3) source credibility. 

Specified and comprehensible 

As suggested by the SEA's frequent pull in of research designed for use, SEA school improvement 
staff preferred research-based strategies or tools or expert advice that offered specific guidance 
for action. Similarly, they sought out research-based articles that were jargon-free and readily 
comprehensible, and that carried a clear message for practice. Such materials were shared with 
educators or assistance providers in the field who were oriented to find solutions to immediate 
concerns, and who may not have had an academic background or proclivity to read such 
materials. An SEA respondent in State C commented: 

I have to be careful when I'm looking at, for example, like a peer review journal, because 
that may be more information than a regular layperson who's not a researcher to make 
sense out of. So, for example, on Monday, we were focusing on the use of data. And so, 
we focused on an article that I found in [ASCD's] Educational Leadership called "The 
New Stupid" by Dr. [Rick] Hess... So, the article is user-friendly and you know, they talk 
about what the research says about data analysis, but it's not so heavy that someone 
who's not a researcher can't get [it]. 

The Hess article explains a study conducted by RAND on why the class-size reduction initiatives 
in California did not yield the same positive outcomes as a class-size initiative in Tennessee. It 
also identifies factors that policymakers should consider when deciding whether to adopt a new, 
research-based initiative, and howto implement an initiative with fidelity to particular principles. 

SEA school improvement staff also sought out evidence presented in forms that enabled them 
to discern key points quickly. They selected data dashboards and graphic data displays to make 
the results quick to identify and more readily understood. Research reviews and summaries, 
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syntheses, or meta-analyses made it easier to acquire a quick scan of the field, such as that 
provided by the What Works Clearinghouse, or the analyses of research in key topics that were 
provided by the RELs or the CACs. 


Contextual validity 

Some respondents considered the internal and external validity of research studies as a factor 
in their use of the information. For example, a staff member in State B noted: "This study, they 
did it with 30 people, okay. This is not enough, to me, to be able to transfer that to a larger 
setting. You know, can this be generalized to ours? Well, okay, they only did it with this one 
group of students. No, that's not what we are looking for." And SEA staff turned to the What 
Works Clearinghouse or research syntheses in part because they evaluated research validity and 
reliability. 

But contextual validity, the notion that an intervention applies, or can be adapted to apply, to 
a particular context (Skinner, 201 2), was a more salient and frequently discussed issue. Several 
respondents in State C talked about the importance of looking at the setting in which the 
research or other evidence was conducted to determine whether it was similar enough to their 
own to be relevant. One SEA staff person, for example, bristled at the proposal that State C 
should adopt a reform model from a state that was much smaller and less diverse than their 
own. They advised local educators to review the settings in which research-based practices had 
been successful, and to judge whether the sites sufficiently resembled their own enough to 
be replicated. Districts and schools were guided to consider the populations that were served, 
as well as the fiscal and human resources that were available to implement the program with 
fidelity to designers' intent. Reflecting on this and similar concerns, a state leader said: 

When I think about information, or theories, or whatever...[l] think about a Venn 
diagram. You know, that place in the middle where theory of practice meets practical 
implementation questions. Where is that place in the middle where... what appears to 
be a good practice in thought, actually works at the ground level, or at the school, or at 
the [district]? 

Similarly, all three states considered how to adjust their interventions in light of varying contexts, 
and of the feasibility of implementation. An SEA leader in State C described this challenge: 

And sometimes [the districts] are not ready for what may be the next step. And 
sometimes we just have to step back and give them a little bit more time. Many of them 
don't have the capacity to do all that we would like for them to do. Either because of a 
will not to, or they really... are so small or too big, like in the big [districts], they have so 
much happening that whatever we ask them to do is just on top of you know, 20 other 


1 46 CONSORTIUM FOR POLICY RESEARCH IN EDUCATION | Cpre.org 


The Use of Knowledge in School Improvement Policy 


programs. And then the opposite effect, in a smaller [district], they don't have. . .the 
personnel or other resources. 

This concept of contextual validity was used to guide adjustments to state strategies and 
supports. As State B became increasingly cognizant of the varying challenges of local contexts, 
they shifted to a selection of supports based on the needs of a particular site. State A created a 
delivery system where staff in different regions of the state could make decisions about how to 
tailor resources and assistance to specific contexts. And, not surprisingly, in deciding whether to 
use a particular research-based strategy or program, SEAs also considered their own capacities. 
As an SEA staff member in State C commented: "We're also very aware of what our resources and 
capabilities are, and we're somewhere at the point of trying to connect all of those in helping to 
refine what we're offering." 

Finally, while contextual validity was used to make some decisions, it was also perceived to be a 
way to convince practitioners of the legitimacy of findings and recommendations. A respondent 
in State C noted that: "One of the problems I have with my schools is you can show them 
research, a good research article and they'll say, 'But this doesn't apply to me because they're not 
like us.' I like information that is presented in a way that my schools can say, 'Hmm, this might 
work. It sounds like us.'" State B sought to insert practice examples from their own schools and 
districts into their communications for similar reasons. 


Source credibility 

The perceived guality of research, other evidence or practitioner knowledge was tightly bound to 
the perceived quality of the source from which it came. The credibility of the source could develop 
through the kind of direct, positive interactions that were necessary to produce relational trust 
(see Chapter 4), or through the reputations that sources gained among peers or educators in the 
field. 

For example, practitioners who were reputed to be successful and well-respected among their 
colleagues were called upon to provide feedback on strategies or tools, serve on advisory groups, 
offer training or coaching, and more. Researchers, or research organizations, also benefited from 
such reputational effects, and their work circulated in the SEAs and among their providers and 
educators. SEA school improvement staff across the three states mentioned similar names, such 
as Robert Marzano, Elizabeth City, Richard Elmore, Richard Stiggins, Sam Redding, and Charlotte 
Danielson. SEA staff often learned about their work from colleagues or local practitioners, and 
these researchers were subsequently invited to present, advise, and provide tools and materials. 

Highly regarded sources could make research-based practices seem more valid and trusted. 
Said one State B staff member: 
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I think, because [Cll is] doing it with schools, state agencies that we are familiar with, 
and we work with, a lot of their [information] comes from those states. So you get to talk 
to the people who are actually involved in it. So, I think that increases, in your mind, the 
validity because you are hearing it, as well as the actual reports. So I think it's a trust 
thing. 

Source reputations could also lend credibility to the states' school improvement strategies and 
generate local educators' confidence. One academic, for example, was invited to help design 
aspects of State C's school improvement work in part because he had been a well-respected 
superintendent, and his involvement was seen as a way to signal the efficacy of the SEA's 
approach to potentially wary district leaders. Similarly, State A sought to hire individuals with 
reputations for being effective in prior work as regional assistance staff. As a senior SEA manager 
explained: 

We needed to set up a way in which districts would look to the people that we were 
hiring as respected and people they would think had something to offer. So this notion 
of bringing in people who were former-had been school district people was something 
that became somewhat central to the design. 

Source reputations could also undermine research findings. Specifically, SEAschool improvement 
staff were aware of the political or ideological reputations of particular sources, and considered 
them when reviewing the findings and methods. When asked whether the information that 
they received was valid and reliable, one State B staff member replied: "reputation has a lot to 
do with it." She perceived, for example, that the Thomas B. Fordham Foundation developed a 
rubric that would assign low grades to extant state standards as part of an agenda to leverage 
broad adoption of the Common Core State Standards, a motive that led herto discounttheir low 
ratings of her state, even though it supported the Common Core. Her perception was bolstered 
when a different organization rated State B standards as better aligned to these new standards. 

The proclivity to rely on familiar and trusted sources, or sources who had gained a positive 
reputation, could enable a kind of limited search. Said an SEA staff member in one of our 
states: "I don't know whether [the office director] finds the best work. I mean, she calls on her 
resources, who have supported her office, and who she trusts over the years, to kind of gather 
what she reads. And in most cases, they're experts, so why not take what they say?" While a few 
comprehensive searches were reported in this SEA, searches were more likely to be conducted in 
this way, and knowledge from these types of searches were more likely to be accessed and used. 

Source reputations could also be generalized to domains that went beyond their original area of 
expertise or research. For example, an SEA director in State C observed: 
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The situation that I kind of find most concerning is when ... we're using a product in 
elementary schools, where at least we know they're based on research, research-based 
practices. So there is evidence for the practices that are embedded. There is evidence for 
the kinds of diagnostics that are in there. ..And then, the developer says, 'Well, we're 
going to build it for middle school and high school.' And we say, 'Great, we'll buy it.' 
And we haven't even seen anything yet. We're not a part of the development. We just 
sort of trust that, because we have a research-based tool in elementary school, and it 
seems to be doing what we want it to do, ...that they will be able to apply that, just sort 
of generalize that to the upper grades, when we know that the needs, say in reading, of 
children in elementary school are very different from the needs of adolescents. 


Other factors 

In addition to the qualities of the evidence (or its sources), other factors influenced the extent 
to which SEA staff sought out and used research, data or practitioner knowledge in their school 
improvement work. Several elements related to the capacity of the SEA as an organization-the 
numbers and types of staff, the resource of time, and the norms or expectations of evidence 
use. The scope of search and use was also related to the availability of relevant research or other 
evidence for action. 

Staff Expertise and Experience 

The extent to which school improvement staff had the background and experience in searching 
for, interpreting, and using research or data varied across the SEAs. As noted in Chapter 2, State B 
had a wide pool of staff with doctorates, while State C had very few. "Most people just don't have 
the background," observed a staff member in State C. Research offices could facilitate search 
and use for school improvement staff, but while the Research Office was very robust in State A, 
State C only had one individual in place, and the SEA had lost that position due to budget cuts 
by the time of our second visit. 

Norms 

As our discussion shows, SEAs frequently sought out RBK to develop or modify their tools and 
strategies, and made references to this evidence in documenting this work. For many, seeking 
out research (and/or turning to those they trusted to have the research-based expertise) was a 
norm of practice. Recall, for example, that States B and C were quite explicit about moving their 
schools and districts away from selecting practices based on anecdote and personal preferences 
to selecting improvement programs with a more proven, research-based record. The SEA in State 
A conducted extensive searches for relevant research, and the research staff helped program 
offices design and implement program evaluations. The SEA's new performance management 
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system is designed to focus staff agency-wide on data-based measures of progress toward 
priority outcomes. Asked to assess the culture of research and data use in their agency, one 
senior member of the Research Department in State A noted: "I think [SEA] folks come to our 
office with more sophisticated questions than they used to, and more frequently come to us 
with questions." Another senior researcher commented: "People are constantly talking about 
research. Both the program coordinators that I work with and the associate commissioners. 
[And] the commissioner." 

State B based its teaching and learning framework directly on the research-based practice guides 
produced by the What Works Clearinghouse, and sought to inculcate norms of research and 
evidence use among practitioners using the resource. Specifically, State B created a website on 
its teaching and learning framework that allowed practitioners to submit instructional practices 
aligned to particular domains. But they required educators to submit their own assessment 
of the scholarship underlying the practice they proposed. Specifically, educators are asked to 
summarize the quality of evidence, how "seminal" the research is, and whether it is confirmed 
by other studies or by experts. The state includes a link to the research evidence next to the 
recommended practice. 

Time 

Time was another organizational resource that influenced the scope and nature of search and 
use. SEA staff needed time to access, read, digest, share, and incorporate evidence, particularly 
research, into their work. Many respondents expressed a desire to know more about new 
research, but found it challenging to find the time to keep current with the literature. Said 
one State B respondent: "What hinders is time. I think all of us would say there are so many 
demands now, so many things that we feel like we want to do, we need to do, and there is just 
not enough time." This respondent was talking about searching for information from internal 
colleagues. The time and effort required to conduct open-ended, external searches is magnified, 
and helps explain why more SEA staff did not engage in those types of searches. One senior 
staff person in State A explained that she simply lacked time to participate in webinars or other 
dissemination activities that could provide useful information. 

I get RELsummaries. I glance atthem. I gettons of webinar information. I'd love to sit 

and do a webinar. [But] we are fast and furious. I'm sure that there's research out there. 

Having the opportunity to read through it and find out what's going on, you know.. ..I 

really depend on my other administrators and their staffs if there is something I need. 

Even when searches were contracted out, it was difficult for SEAs to manage and maintain the 
commitment to engage in these collaborations. One staff member from State B bemoaned: 
"There are some things I would like to have more time, working with [a national research 
organization], . .. But there's only so much time." 
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Supply 

Beyond the organizational factors, search was also influenced simply by the availability of relevant 
evidence. SEA staff lamented that rigorous evidence for action was sometimes unavailable, or 
found that it was very difficult to attribute to specific policy strategies. The Research Department 
in State A, for example, reported it had difficulty finding research on effective ways to work with 
districts, rather than schools, when their state was updating its SIF. As one of its staff noted: "It 
didn't take us long [to look] because there wasn't much out there." Respondents also reported 
that sometimes it was necessary to act without research guidance in order to innovate. One staff 
member from State B noted: "We have some strategies out there that are relatively so new that 
we don't have a large base of research on them. We just don't know." Echoed another from State 
C: "I think that's something we all forget in the research community. You have to experiment 
before you know if it's going to work." But they pressed for evidence. State C pressed districts 
to evaluate their progress, and more systematically collect evidence of effects. State A evaluated 
programs and turned to its internal monitoring reports and data to identify elements of effective 
school practice. 

Conclusion 

These states provide us with ample evidence thatthey engage in active guests to embed research 
and other forms of knowledge in their school improvement practices. This finding makes no 
claims about the efficacy of these endeavors, or the depth of individuals' understanding of any 
particular set of ideas. But SEA staff do attend to research-particularly research designed for use, 
summaries and syntheses-along with other evidence if it is readily accessible, understandable 
and trusted, and addresses the pragmatic issues that they deal with day to day: the scarcity 
of time; the need to act guickly in response to external policy demands and accountability 
pressures; and, the need to communicate to those who may not have the background or 
inclination or time to delve into lengthy, academic writing. These SEAs had organizational 
cultures and staff that, to varying degrees, supported the acguisition and use of research and 
other evidence. They used different mechanisms to acguire and incorporate this information. 
And use was not self-actualizing. The information was translated or revised to suit the context 
or the prior convictions of SEA staff. Earlier chapters discussed the processes by which the 
incorporation of knowledge occurred, and indicated that highly prominent sources-those who 
were freguently sought out for information-were also typically individuals who often engaged 
with staff and helped them make sense of the information for their practice. Internal knowledge 
brokers and external sources like Cll performed these functions, and trust in the guality of their 
knowledge was generalized. 

Finally, as researchers studying research use, we also want to share our struggle to apply other 
constructs developed by those studying this question. We tried to use a conceptual frame that 
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categorized knowledge use as conceptual, instrumental, political, or imposed (Weiss et al., 
2008). We ultimately determined that we could not reliably apply these categories because 
they depended heavily upon the perceptions of the respondent, and, perhaps, the observer. 
For example, whether use is political often depends on where you sit in the chain of decision- 
making. As we noted above, State B considered Fordham's review of their alignment with the 
Common Core as political, but Fordham researchers would argue that their criteria are the best 
objective measures. We also described several examples where an SEA inserted references to 
research in their school improvement documents (instrumental use) as a way to signal to their 
users (such as schools and districts) a body of knowledge that would build some conceptual 
understanding or legitimacy for their initiatives. In other words, the categorization of use is a 
very fluid and position- and level-specific, as well as temporal, construct. 
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Summary and Implications 


Over the last 20 years, SEAs have been given considerably more responsibilities for directing 
and guiding the improvement of low-performing schools. At the same time, SEAs and school 
districts have been pressed to incorporate research-based school improvement strategies into 
their systems of support for these schools. While researchers have examined how schools and 
districts use research, we know little about how SEAs search for, select, and use research and 
other kinds of evidence, especially in their school improvement policies and programs. 

This exploratory study was designed to address that gap. Because we focused on research use 
for school improvement in only three states, our findings may not generalize to other SEAs or to 
other education policy areas. In addition, we did not examine the effects of SEAs use of research 
or other evidence on improving school or student outcomes. This study is, however, the first to 
systematically map information networks within SEAs and between SEAs and external sources of 
support, and provides important insights into how SEA staff search for and incorporate research 
into their work. In this chapter, we summarize our major findings and discuss implications for 
policy and future research. 

Major Findings 

Multiple staff in the three study states actively searched for and were receptive to research and 
related information on school improvement from both within and outside their agencies. About 
three-guarters of staff in each agency asked their SEA colleagues for research advice, while a 
little less than one-third turned to external organizations or individuals for similar information. 
In each SEA, some, but not all, of these staff named multiple colleagues, offices, and external 
organizations as sources of research information. 

Contrary to the usual image of SEAs as siloed organizations, and although the three SEAs in 
the study remained organized largely by function, we found considerable cross-office and 
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department communication. Multiple factors stimulated this communication across silos, 
including state and federal accountability demands, competition for federal grants that required 
integrated proposals, reduced SEA staffing, and SEA leaders committed to more collaborative 
organizational cultures. All three SEAs held regular meetings with senior staff from across their 
organizations, and created ad hoc cross-agency teams to respond to specific policy issues. While 
these broad cross-department connections facilitated the flow of information and new ideas, 
they were weaker than connections within departments. In other words, interactions between 
colleagues in the same department or office were more frequent, and were viewed as having more 
influence on participants' work, than cross-department connections. Thus, formal organizational 
structures still delineate many functional responsibilities and lines of communication. 

While the school improvement research networks in each SEA included an array of staff 
from across its organization and numerous external organizations, only a small number of 
individuals, offices and external partners were central actors in these networks. In two states, the 
directors of school improvement were the most influential and well-connected SEA staff in their 
research networks. Other key participants included leadership from the research, curriculum 
and instruction, assessment and accountability, ESEA program monitoring and, in one of these 
states, special education offices. Influence over research and knowledge brokering tended to be 
concentrated in formal positions-among directors and managers-in these two SEAs. While staff 
at all levels of the organization searched for information outside the SEA, they were less likely to 
seek information from lower level staff, some of whom conducted broad searches. Thus the flow 
of knowledge within the agency was somewhat constrained and some knowledge or expertise 
remained disconnected from the key users of research. In the third SEA, where our analysis 
was limited to the office level, a relatively large research office was at the center of its research 
network, with multiple connections to most others departments in its agency. Other highly 
influential offices included school improvement, accountability, and curriculum and instruction. 
External organizations tended to be located at the periphery of the research networks in all three 
states, with a very small number playing a key role in two of the SEAs. 

A small set of influential knowledge brokers who conducted broad searches brought research 
and other kinds of information from these different sources into core networks that collectively 
addressed problems of school improvement. These groups, which generally included leaders 
and other staff within school improvement offices, a few key external organizations and, in a 
few instances, colleagues from other departments in the agency, incorporated research into 
their school improvement strategies through a distinctly social process in which the network 
members interpreted, challenged, and otherwise made sense of research over time. During 
this incorporation process, the core network groups used local practitioners' feedback, 
state professionals' experience, and external partners' knowledge of relevant research to 
contextualize various research findings in light of their states' school improvement needs. In 
contrast to models of research dissemination in which generalized, primarily decontextualized 
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findings advanced by researchers are transmitted to users, in these core networks users and a 
few external providers worked collectively to adapt research to address particular problems and, 
in some cases, to co-construct new useable knowledge for guiding action. 

Social networks, especially the stronger core networks, enabled SEAs to manage three long- 
standing problems that have been obstacles to the use of research in many organizations: 
matching research to pressing school improvementorothersocial problems; translating research 
into guides for program development and implementation; and addressing research overload. 
The core network groups created the collective capacity needed to sift through and interpret an 
ever growing body of research; developed a body of shared knowledge and resources, based 
on research principles, that was embedded in school improvement frameworks and guidance 
materials; and supported more technical alignment and conceptual coherence in the states' 
school improvement delivery systems and across offices in the SEAs. Overlapping network 
structures that connected key SEA staff with practitioners allowed for an implementation, 
feedback and revision loop that kept school improvement strategies dynamic and responsive. 
Finally, through a collective problem-solving process, networks enabled members to benefit from 
the social capital in the stronger core network relationships. SEA staff perceived information, 
decision-making, evolving improvement strategies, and ideas to be more trustworthy and 
efficacious within the context of these structures. 

Although fewer SEA staff turned to external than internal sources of research and other 
kinds of evidence, external organizations played a key role in the research and practitioner 
knowledge that policymakers accessed and used. Among the large and diverse array of external 
organizations named by SEA staff, federal agencies, federally-funded research and technical 
assistance centers, and national professional membership organizations were major sources 
for research information. Members of the federal comprehensive assistance center system, for 
example, were key actors in the school improvement networks in two of the SEAs in our study, 
helping SEA staff identify timely, relevant research or potential research-based solutions to their 
problems. Fewer staff sought research from universities or research organizations. Practitioner 
knowledge was more often acguired through in-state organizations, such as school districts, area 
education agencies, and state level professional associations. 

Staff in the three SEAs turned to a different mix of external sources for both research and 
practitioner advice. This variation reflected difference in stages of policy development, internal 
capacity, the structure of external environments in the states, and prior partnership histories. 
All of the SEAs, however, turned to external sources that were familiar and trusted, and that 
they believed provided credibility to their work and validated that their approach to school 
improvement was in line with other states' policy or practice. External organizations had the 
flexibility and the resources to address problems or conduct work that the SEAs could not. 
Finally, they provided easy access to research, and played an important role in synthesizing and 
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packaging research to make it useable, and useful, to SEA work. The more influential external 
providers in SEAs networks brokered, jointly developed with states, or helped states adapt 
research-based, but useable tools that translated research into more specified guides for action. 

The SEAs in our study valued and used research to inform the design of their school 
improvement frameworks, processes, tools, and other forms of school improvement assistance. 
Traditional forms of research, primarily packaged and presented in short summaries, syntheses, 
or meta-analyses, were included most extensively in the development and revision of school 
improvement frameworks. This research was also used to validate states' school improvement 
strategies, legitimize solutions, and develop shared language and understanding throughout 
the system. States also created their own original research by undertaking both formal and 
informal evaluations, often contracting with research organizations to conduct these studies. 
But SEAs in the study more often relied on research-based tools and strategies that provided very 
specific and concrete guides to action, what we call "research designed for use;" that is, research 
that is actionable, feasible and addresses a pressing problem of policy or practice. Even this 
form of research was sometimes further adapted and modified to the specific policies or context 
of each state, often in collaboration with external organizations and local practitioners. 

SEA staff turned to research that was specified and understandable, and that they felt had 
contextual validity and source credibility. Research use was also influenced by several elements 
related to the capacity of the SEA: staff expertise and experience in searching for, interpreting 
and using research; the norms or expectations of evidence use; and time to access, read, digest, 
share and incorporate evidence into their work. Research use was also affected by the availability 
of relevant information. 

Policy Implications 

These findings shed light on ways that SEAs and policymakers can strengthen research-based 
knowledge use in their organizations. First, SEAs should draw on the infrastructure outside their 
boundaries, such as technical assistance centers, state and national professional membership 
organizations, other professional networks, and universities to access research and research 
designed for use. External organizations are not only a source of research and new ideas, but 
play important roles in synthesizing research and helping SEA staff apply and adapt research 
to their problems of practice. They also have the potential to expand the limited capacity of 
over-burdened SEAs. Reaching out to external providers, however, will require SEAs to develop 
a culture of research use, and to build their capacity to broker research search and incorporation 
and to assess the underlying quality of the research and research designed for use produced by 
these organizations. In addition, our study showed that SEAs may be missing some important 
knowledge resources from little known sources inside their organizations. By cultivating 
multiple knowledge brokers within the SEA to access and circulate a diverse array of research, 
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SEA knowledge networks will have access to a broader range of expertise and could also be less 
vulnerable to staff turnover. 

Second, SEAs should also identify, connect and nurture knowledge brokers in their agencies and 
in external organizations who work on common problems. Fostering working groups composed 
of influential SEA brokers, key research sources, and practitioners to adapt generalized findings 
into more useable information in the context of particular state problems can facilitate the 
incorporation of research into policy and practice. Cultivating professional knowledge networks 
in configurations that cut across SEA boundaries may also develop internal expertise and 
collective capacity, support alignment and coherence of policies, and facilitate a continuous 
improvement cycle for SEA policies and practices. SEA staff could develop and lead ongoing 
networks involving research organizations, practitioners, and their own staff to solve specific 
problems and advance state policy. The New York State Education Department's Education 
Finance Research Consortium provides one example of this approach. 

Third, policymakers should encourage and support SEA evaluations of their own programs. 
These evaluations, particularly of the implementation of school improvement programs, provide 
critical, systemic feedback to agency staff. But SEAs often lack the human resources to design 
these studies and the fiscal resources to conduct them. 

Research Implications 

This study also has several implications forfurther research. First, because this was an exploratory 
study of knowledge use in school improvement policy by a small, purposive sample of SEAs, 
researchers should study the use of research and other types of evidence in additional SEAs and 
in other education policy areas to see whether the findings reported here generalize to other 
SEA settings. 

Second, the implications of our study for improving the quality of school improvement policies 
are indirect. Our findings are based on the perspective of SEA staff, the external providers they 
used, and indicators from other studies, not from an analysis of the effects of these policies 
on schools. Thus, further research should examine connections among patterns of internal 
or external information flow within an SEA, the number and type of information sources, the 
types of evidence people access and use in decision-making, and consequences for policy and 
practice. Likewise, future research should track how the integrity of research is maintained 
during implementation to understand the tradeoffs between fidelity to models and adaptation 
to contextual contingencies. 

Third, we see a need to assess the quality of research acquired by SEA staff and underlying 
"research designed for use" products. While many of these products were written by or cited 
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national experts, sometimes research was added in a fairly superficial manner, and some of the 
underlying studies were weak. 

Finally, the knowledge base needs to be strengthened, which includes supporting more varied 
types of research on policy implementation and effects and on understudied areas of education 
policy. Such evaluations should be exchanged across states to improve more systematic learning 
about implementation, an area that SEA staff want but often have little more than anecdotal 
exchanges on. SEA knowledge use is only as good as the knowledge base. 
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Table 1 .1 : List of Major Codes 


1. Political/lnstitutional/Contextual Factors 

a. SEA organizational structure 

b. School improvement delivery system 

c. Policy 

d. Human capital 

e. Political context/issues 

f. Funding 

g. Organizational culture 

2. Policy Area 

a. School improvement 

b. Accountability/assessment 

c. Curriculum and instruction 

d. Teacher quality 

e. Other 

3. Communication/Collaboration 

a. Origin 

b. Structure 

i. Hierarchical 

ii. Lateral 

iii. Internal 

iv. External 

c. Strength 

d. Qualities 

4. Knowledge Description 


a. Type 
i. 

RBK 

ii. 

EBK 

iii. 

Practitioner 

iv. 

Unclear 

b. Form 
i. 

Designed for use 

ii. 

Scaffolding/support for use 


5. Sources (for Knowledge/Information) 

a. SEA 

b. LEA 

c. Associations 

d. Research firms 

e. Government(otherthan SEA and LEA) 

f. Institutions of higher education 

g. Other 

6. Search and Incorporation 

a. Problem definition/goals/decision-mak- 
ing/problem-solving/process 

b. Factors in knowledge use 

c. Ongoing improvement/feedback 

7. Knowledge Use 

8. Coherence 
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Table 1.2: Survey Coverage and Response Rates, by State and 
Knowledge Network 



State A 

State B 

State C 

Total 

# of surveys administered 

171 

245 

194 

610 

#of completed surveys 

111 

177 

162 

450 

Response rate 

64.9% 

72.2% 

83.5% 

73.8% 

# of respondents self-identifying as involved in school 
improvement work 

97 

127 

81 

305 

# of respondents answering work network questions 

97 

123 

80 

300 

# of respondents answering research network 
questions 

95 

121 

79 

295 

# of respondents answering data network questions 

90 

118 

79 

287 

# of respondents answering practitioner network 
questions 

89 

115 

78 

282 
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Table 1.3: Trust and Efficacy Questions 


Please mark the extent to which you agree with the following statements for the individuals (or the offices 
and organizations) identified below. 

(Scale: 1=Strongly Disagree; 2=Disagree; 3=Slightly Disagree; 4=Slightly Agree; 5=Ag ree; 6=Strongly 
Agree) 

a. The group of individuals above can be counted on to respect the opinions and 
information provided by others. 

b. The group of individuals above can be trusted to do their best, no matter how 
challenging the task. 

c. The group of individuals above consistently provides information and advice that 
is valid and reliable. 

d. The group of individuals above can be counted on to keep their commitments to 
the work we do together. 

e. The group of individuals above can has the expertise to find and apply the 
appropriate research to improve our programs and practices. 

f. The group of individuals above is skilled at interpreting and using many forms of 
research to improve our programs and practices. 

g. The group of individuals above and I have the staffing and monetary resources to 
search for and use research to improve our programs and practices. 

h. The group of individuals above and I are confident that the research we use will 
enable us to successfully improve low performing schools and schools and school 
districts. 

i. The political environment in this state makes it easy for the group of individuals 
above and me to incorporate research into our work. 

Calculation of Trust and Efficacy Scores 

Separate average scores were calculated for each respondent for the four trust statements (a-d) and for the five 
efficacy statements (e-i). The average trust scores from all respondents were then averaged to arrive at an overall 
average trust score for each network. The same process was used to arrive at average efficacy scores. 

Respondents were asked to respond to these trust and efficacy statements in regards to all of the individuals (or 
offices and organizations) they had named fora particular network. Therefore, their responses reflect their feelings 
about the group, not necessarily about each individual. Each respondent also rated his/her own selected group of 
individuals.Therefore, the average trust and efficacy scores for a network do not reflect a set group of individuals, 
but rather the group of individuals selected by each of the respondents. 
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Tablel .4: Calculation of Strength of Network Connections 



Daily or almost 
daily (200) 

Once or twice per 
week (70) 

Once or twice per 
month (20) 

A few times a year 
(5) 

Highly influential 
(100%) 

200 

70 

20 

5.0 

Influential (70%) 

140 

49 

14 

3.5 

Somewhat 
influential (40%) 

80 

28 

8 

2.0 

Not very 

influential (10%) 

20 

7 

2 

0.5 
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Table 3.1 : Number and Strength of Within and Across SEA Ties and Ties to External 
Organizations, by Network and State 


Network 

Percent of respondents with 
out-ties to departments 
within the SEA 

Number of SEA ties within 
and across departments 

Numberof within 
department out-ties 

Average strength of within 
department out-ties 

Number of cross department 
out-ties 

Average strength of cross 
department out-ties 

Number of cross department 
in-ties 

Average strength of cross 
department in-ties 

Percent of respondents with 
out-ties to organizations 
outside of the SEA 

Numberof out-ties to 
external organizations 

Average strength of external 
out-ties 



State A 

94.56% 

999 

221 

119.46 

778 

37.03 

778 

37.03 

na 

na 

na 

State B 

73.68% 

784 

314 

114.87 

456 

26.67 

456 

26.67 

na 

na 

na 

State C 

90.14% 

819 

223 

124.24 

432 

27.82 

432 

27.82 

na 

na 

na 

RBK 

State A 

79.12% 

419 

114 

71.04 

305 

28.07 

305 

28.07 

25.27% 

59 

31.96 

State B 

54.03% 

402 

168 

75.58 

234 

23.58 

234 

23.58 

28.00% 

77 

30.87 

State C 

65.38% 

264 

108 

92.3 

156 

27.86 

156 

27.86 

30.76% 

65 

37.5 

DATA 

State A 

75.86% 

391 

106 

77.77 

285 

26.94 

285 

26.94 

10.34% 

21 

25.76 

State B 

69.35% 

361 

170 

80.74 

191 

16.85 

191 

16.85 

18.00% 

37 

45.15 

State C 

65.06% 

259 

93 

97.29 

166 

25.76 

166 

25.76 

13.25% 

21 

35.67 

PRAC 

State A 

68.83% 

275 

82 

68.51 

193 

28.88 

193 

28.88 

31.17% 

46 

31.57 

State B 

26.02% 

191 

113 

75.49 

78 

23.06 

78 

23.06 

28.00% 

62 

26.22 

State C 

50.00% 

150 

76 

102.3 

74 

40.45 

74 

40.45 

17.94% 

33 

30.44 
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Figure 3.1: State A, Data Network, ties only 
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Figure 3.2: State A Data Network, strength of ties 
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Figure 3.3: State B f Data Network, ties only 
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Figure 3.4: State B f Data Network, strength of ties 
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Figure 3.5: State C f Data Network, ties only 
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Figure 3.6: State C, Data Network, strength of ties 
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Table 5.1: Average Strength* of Connections 



Within Silo 
Out-ties 

Cross-Silo 

Out-ties 

External 

Out-ties 

STATE A 

Research 

71.04 

28.07 

31.96 

Data 

77.77 

26.94 

25.76 

Practitioner 

68.51 

28.88 

31.57 

STATE B 

Research 

75.58 

23.58 

30.87 

Data 

80.74 

16.85 

45.15 

Practitioner 

75.49 

23.06 

26.22 

STATE C 

Research 

92.30 

27.86 

37.50 

Data 

97.29 

25.76 

35.67 

Practitioner 

102.30 

40.45 

30.44 


*Strength is a composite score of frequency of communication times influence. See 
Chapter 3 for a more complete discussion and set of figures regarding this analysis. 
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Table 5.2: Number and Percent of External Mentions, by 
Organizational Sector, Knowledge Network, and State 


Network 

Research 

Data 

Practitioner 

STATE A 

Federal Government 

19(32%) 

0 

2 (4%) 

State/Reqional/Local Government 

4(7%) 

6(29%) 

21 (46%) 

Professional Membership 

6(10%) 

10(48%) 

3 (7%) 

IHE 

6(10%) 

0 

7(15%) 

Research Organization 

6(10%) 

2(10%) 

1 (2%) 

Provider Organization 

3 ( 5%) 

0 

6(13%) 

Education Journal 

6(10%) 

0 

0 

Other 

5 (9%) 

3(14%) 

1 (2%) 

Unknown 

4(7%) 

0 

5(11%) 

TOTAL 

59 

21 

46 

STATE B 

Federal Government 

29 (36%) 

8(21%) 

6(9%) 

State/Reqional/Local Government 

2 (3%) 

13(33%) 

18(26%) 

Professional Membership 

26 (33%) 

11(28%) 

29 (43%) 

IHE 

6 (8%) 

1 (3%) 

5 (7%) 

Research Organization 

7 (9%) 

2 (5%) 

2 (3%) 

Provider Organization 

2 (3%) 

2 (5%) 

3 (4%) 

Education Journal 

4(5%) 

1 (3%) 

2 (3%) 

Other 

4(5%) 

1 (3%) 

0 

Unknown 

0 

0 

3 (4%) 

TOTAL 

80 

39 

68 

STATE C 

Federal Government 

27 (42%) 

8 (32%) 

7(21%) 

State/Reqional/Local Government 

1 (2%) 

3(12%) 

0 

Professional Membership 

20(31%) 

8 (32%) 

16(47%) 

IHE 

12(19%) 

4(16%) 

8(24%) 

Research Organization 

0 

0 

0 

Provider Organization 

0 

0 

0 

Education Journal 

0 

0 

0 

Other 

3 (5%) 

0 

0 

Unknown 

1 (2%) 

2 (8%) 

3 (9%) 

TOTAL 

64 

25 

34 
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Table 5.3: Number and Percent of Federal Government 
Mentions for Research Knowledge, by Source and State 



Federal Agencies 

NCLB CACs 

RELs 

Other Federal 

Total 


# 

% 

# 

% 

# 

% 

# 

% 

# 

State A 

7 

37% 

2 

11% 

7 

37% 

3 

16% 

19 

State B 

13 

45% 

10 

35% 

1 

3% 

5 

17% 

29 

State C 

9 

33% 

11 

41% 

2 

7% 

5 

19% 

27 
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Table 5.4: Professional Membership Organizations, by Level 
Type, and State 

Research Network 





National PMAs 

State PMAs 

Occupational 

PMAs 

Subject Matter 
PMAs 


PMAs 

# mentions 

# mentions 

% mentions 

# mentions 

% mentions 

# mentions 

% mentions 

# mentions 

% mentions 

State A 

4 

6 

6 

100.0% 

0 

0 

3 

50.0% 

3 

50.0% 

State B 

15 

26 

20 

76.9% 

6 

23.1% 

18 

69.2% 

8 

30.8% 

State C 

14 

20 

18 

90.0% 

2 

10.0% 

16 

80.0% 

4 

20.0% 

Total 

33 

52 

44 

84.6% 

8 

15.4% 

37 

71.2% 

15 

28.8% 


Practitioner Network 
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Table 5.5: Distinct Within-State External Organizations, by 
Sector and Knowledge Network within Each State 



State A 

State B 

State C 


Total 

RBK 

Data 

PK 

Total 

RBK 

Data 

PK 

Total 

RBK 

Data 

PK 

Govt: 

State 

4 ( 17 %) 

2 ( 25 %) 

3 ( 27 %) 

3 ( 20 %) 

4 ( 13 %) 

0 

1 ( 8 %) 

2 ( 9 %) 

3 ( 23 %) 

0 

2 ( 33 %) 

1 ( 14 %) 

Govt: 

LEA 

7 ( 29 %) 

1 ( 13 %) 

1 ( 9 %) 

7 ( 47 %) 

2 ( 7 %) 

0 

2 ( 15 %) 

2 ( 9 %) 

0 

0 

0 

0 

Govt: 

Regional 

0 

0 

0 

0 

5 ( 17 %) 

0 

4 ( 31 %) 

4 ( 17 %) 

0 

0 

0 

0 

PMA 

5 ( 21 %) 

0 

5 ( 45 %) 

1 ( 7 %) 

16 ( 53 %) 

4 ( 67 %) 

4 ( 31 %) 

14 ( 61 %) 

4 ( 31 %) 

2 ( 25 %) 

1 ( 17 %) 

4 ( 57 %) 

IHE 

4 ( 17 %) 

3 ( 37 %) 

0 

3 ( 20 %) 

2 ( 7 %) 

2 ( 33 %) 

1 ( 8 %) 

1 ( 4 %) 

6 ( 46 %) 

6 ( 75 %) 

3 ( 50 %) 

2 ( 29 %) 

Provider 

1 ( 4 %) 

0 

0 

1 ( 7 %) 

0 

0 

0 

0 

0 

0 

0 

0 

Research 

1 ( 4 %) 

1 ( 13 %) 

1 ( 9 %) 

0 

1 ( 3 %) 

0 

1 ( 8 %) 

0 

0 

0 

0 

0 

Advocacy/ 

Fdn. 

2 ( 8 %) 

1 ( 13 %) 

1 ( 9 %) 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Total 

24 

8 

11 

15 

30 

6 

13 

23 

13 

8 

6 

7 
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Table 6.1: Types and Forms of Knowledge Used in Work, by 
Frequency and State 

How often do you use the following to inform your work? 


State A 

Never 

A Few 
Times a 
Year 

Once or 
Twice a 
Month 

Once or 
Twice a 
Week 

Daily or 
Almost 
Daily 

Research (e.g., published original 
research, research syntheses or 
summaries, meta-analyses) 

4.2% 

31.3% 

27.1% 

20.8% 

16.7% 

Program evaluation (e.g., published or 
unpublished evaluations of programs or 
policies) 

8.3% 

43.8% 

28.1% 

12.5% 

7.3% 

Research-based guidance (e.g. books, 
best practice guides, protocols or other 
tools written for practitioners) 

12.4% 

28.9% 

24.7% 

23.7% 

10.3% 

Data on schools in your state (e.g., 
student outcome data, administrative 
data such as the distribution of highly 
qualified teachers, etc.) 

3.1% 

14.4% 

17.5% 

28.9% 

36.1% 

Advice from colleagues within the SEA 

0.00% 

4.1% 

12.4% 

30.9% 

52.6% 

Advice from practitioners outside of the 
SEA who have knowledge of how your 
programs work within your state or in 
other states 

11.3% 

19.6% 

29.9% 

29.9% 

9.3% 
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Table 6.1: Types and Forms of Knowledge Used in Work, by 
Frequency and State (continued) 


State B 

Never 

A Few 
Times A 
Year 

Once or 
Twice a 
Month 

Once or 
Twice a 
Week 

Daily or 
Almost 
Daily 

Research (e.g., published original 
research, research syntheses or 
summaries, meta-analyses) 

7.9% 

18.3% 

22.2% 

35.7% 

15.9% 

Program evaluation (e.g., published or 
unpublished evaluations of programs or 
policies) 

7.9% 

31.0% 

23.0% 

27.0% 

11.1% 

Research-based guidance (e.g. books, 
best practice guides, protocols or other 
tools written for practitioners) 

7.3% 

19.5% 

18.7% 

25.2% 

29.3% 

Data on schools in your state (e.g., 
student outcome data, administrative 
data such as the distribution of highly 
qualified teachers, etc.) 

0.0% 

15.2% 

23.2% 

28.0% 

33.6% 

Advice from colleagues within the SEA 

1.6% 

4.8% 

11.2% 

39.2% 

43.2% 

Advice from practitioners outside of the 
SEA who have knowledge of how your 
programs work within your state or in 
other states 

6.4% 

23.8% 

24.6% 

35.7% 

9.5% 
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Table 6.1: Types and Forms of Knowledge Used in Work, by 
Frequency and State (continued) 


State C 

Never 

A Few 
Times a 
Year 

Once or 
Twice a 
Month 

Once or 
Twice a 
Week 

Daily or 
Almost 
Daily 

Research (e.g., published original 
research, research syntheses or 
summaries, meta-analyses) 

13.8% 

23.8% 

23.8% 

21.3% 

17.5% 

Program evaluation (e.g., published or 
unpublished evaluations of programs or 
policies) 

12.7% 

34.2% 

29.1% 

20.3% 

3.8% 

Research-based guidance (e.g. books, 
best practice guides, protocols or other 
tools written for practitioners) 

13.9% 

17.7% 

24.1% 

24.1% 

20.3% 

Data on schools in your state (e.g., 
student outcome data, administrative 
data such as the distribution of highly 
qualified teachers, etc.) 

6.2% 

16.1% 

23.5% 

23.5% 

32.1% 

Advice from colleagues within the SEA 

6.4% 

5.1% 

12.8% 

12.8% 

52.6% 

Advice from practitioners outside of the 
SEA who have knowledge of how your 
programs work within your state or in 
other states 

11.3% 

23.8% 

32.5% 

23.8% 

8.8% 
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